2024-09-16T00:50:38.111 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T00:50:38.138 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T00:50:38.232 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906435' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 8921c8ef947807b717db041d85f0761b7914d5fc subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyEjef7fHSp8/4YbrT71uxr9e6HSK6aZEdwdgt6woksngmQ2CmnKdoI0IBmiKxz54Ao3tqk/sOKzqpi8Z1sUtA= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNB9cwwxV1BJKQxPUD5b0vny4PqzZcC4fdzoMBgnStvZUCFCOPNiNFFBnweqL+6eh9HAl4axIO+cKf4zp2rGeDc= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF3aPuaFP2L05jKAGd67bn98jO0zS/Pw1SxRnbrvdGT7i4sRiuF8xDQ+5kjocM8SHePsZMrI+IeG9vJnimWhuTU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T00:50:38.232 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T00:50:38.233 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T00:50:38.233 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T00:50:38.234 INFO:teuthology.task.internal:Checking packages... 2024-09-16T00:50:38.258 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T00:50:38.258 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T00:50:38.258 INFO:teuthology.packaging:ref: None 2024-09-16T00:50:38.258 INFO:teuthology.packaging:tag: None 2024-09-16T00:50:38.258 INFO:teuthology.packaging:branch: squid 2024-09-16T00:50:38.258 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:50:38.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T00:50:38.476 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T00:50:38.479 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T00:50:38.485 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T00:50:38.486 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T00:50:38.503 INFO:teuthology.task.internal:Saving configuration 2024-09-16T00:50:38.537 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T00:50:38.547 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T00:50:38.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 00:45:38.059559', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyEjef7fHSp8/4YbrT71uxr9e6HSK6aZEdwdgt6woksngmQ2CmnKdoI0IBmiKxz54Ao3tqk/sOKzqpi8Z1sUtA='} 2024-09-16T00:50:38.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 00:45:38.061223', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNB9cwwxV1BJKQxPUD5b0vny4PqzZcC4fdzoMBgnStvZUCFCOPNiNFFBnweqL+6eh9HAl4axIO+cKf4zp2rGeDc='} 2024-09-16T00:50:38.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 00:45:38.057592', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF3aPuaFP2L05jKAGd67bn98jO0zS/Pw1SxRnbrvdGT7i4sRiuF8xDQ+5kjocM8SHePsZMrI+IeG9vJnimWhuTU='} 2024-09-16T00:50:38.614 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T00:50:38.624 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-16T00:50:38.625 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-16T00:50:38.625 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-16T00:50:38.625 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T00:50:38.718 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f574bf67be0>, signals=[15]) 2024-09-16T00:50:38.718 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T00:50:38.725 INFO:teuthology.task.internal:Opening connections... 2024-09-16T00:50:38.726 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T00:50:38.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:50:38.803 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T00:50:38.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:50:38.879 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-09-16T00:50:38.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:50:38.957 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T00:50:38.964 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-09-16T00:50:38.983 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-09-16T00:50:38.983 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-09-16T00:50:39.042 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-09-16T00:50:39.043 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T00:50:39.043 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-09-16T00:50:39.043 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T00:50:39.043 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T00:50:39.043 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T00:50:39.044 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-09-16T00:50:39.072 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-09-16T00:50:39.091 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-09-16T00:50:39.092 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2024-09-16T00:50:39.150 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T00:50:39.151 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T00:50:39.151 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-09-16T00:50:39.172 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-09-16T00:50:39.191 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-09-16T00:50:39.191 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-09-16T00:50:39.250 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-09-16T00:50:39.250 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-09-16T00:50:39.250 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T00:50:39.251 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-09-16T00:50:39.252 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T00:50:39.252 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T00:50:39.252 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T00:50:39.252 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-09-16T00:50:39.272 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T00:50:39.287 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T00:50:39.326 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T00:50:39.326 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-09-16T00:50:39.329 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-09-16T00:50:39.332 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-09-16T00:50:39.349 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T00:50:39.360 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T00:50:39.360 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-09-16T00:50:39.387 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-09-16T00:50:39.390 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-09-16T00:50:39.500 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T00:50:39.587 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T00:50:39.605 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-16T00:50:39.605 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-16T00:50:39.606 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-16T00:50:39.606 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T00:50:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:50:39.624 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-16T00:50:39.682 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-503.el9.x86_64 2024-09-16T00:50:39.683 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-503.el9.x86_64 2024-09-16T00:50:39.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-09-16T00:50:42.118 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-16T00:50:44.419 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-16T00:50:48.753 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-16T00:50:50.379 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-16T00:50:51.159 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-16T00:50:56.706 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T00:50:58.737 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T00:50:58.737 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T00:50:58.823 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-16T00:50:58.842 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-16T00:50:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T00:50:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 79 M 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 126 M 2024-09-16T00:50:58.844 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-16T00:50:59.129 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 80: Network is unreachable] 2024-09-16T00:50:59.129 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-core-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 80: Network is unreachable] 2024-09-16T00:50:59.129 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-modules-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 80: Network is unreachable] 2024-09-16T00:50:59.133 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-modules-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 443: Network is unreachable] 2024-09-16T00:50:59.149 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 443: Network is unreachable] 2024-09-16T00:50:59.150 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-core-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 443: Network is unreachable] 2024-09-16T00:50:59.307 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 123 kB/s | 23 kB 00:00 2024-09-16T00:50:59.459 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-modules-core-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 443: Network is unreachable] 2024-09-16T00:50:59.477 INFO:teuthology.orchestra.run.smithi078.stdout:[MIRROR] kernel-modules-core-5.14.0-505.el9.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-505.el9.x86_64.rpm [Failed to connect to mirror.facebook.net port 80: Network is unreachable] 2024-09-16T00:51:03.665 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-09-16T00:51:06.207 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.1 MB/s | 29 MB 00:06 2024-09-16T00:51:13.648 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.4 MB/s | 35 MB 00:14 2024-09-16T00:51:13.648 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:51:13.648 INFO:teuthology.orchestra.run.smithi078.stdout:Total 5.3 MB/s | 79 MB 00:14 2024-09-16T00:51:14.329 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-16T00:51:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-16T00:51:14.687 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-16T00:51:15.648 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-16T00:51:15.649 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-16T00:51:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-16T00:51:18.734 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T00:51:18.816 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T00:51:21.736 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T00:51:21.870 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T00:51:26.679 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T00:51:26.711 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:51:31.569 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:52:09.915 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:52:09.956 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:52:10.330 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:52:10.331 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T00:52:10.331 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T00:52:10.331 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T00:52:11.190 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T00:52:11.190 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:52:11.190 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:52:11.191 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-16T00:52:11.581 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T00:52:11.661 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T00:52:11.662 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T00:52:11.662 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T00:52:11.662 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T00:52:11.662 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-16T00:52:11.662 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T00:52:11.663 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T00:52:11.722 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T00:52:11.722 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T00:52:11.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 82.el9 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-09-16T00:52:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8274670 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-09-16T00:52:11.755 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:52:11.756 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T00:52:11.757 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T00:52:11.757 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-505.el9.x86_64 2024-09-16T00:52:11.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T00:52:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-09-16T00:52:13.704 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T00:52:13.737 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-09-16T00:52:13.739 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-16T00:52:13.755 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.pSXvDYyjNt 2024-09-16T00:52:13.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pSXvDYyjNt 2024-09-16T00:52:13.822 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.pSXvDYyjNt 2024-09-16T00:52:13.966 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.pSXvDYyjNt is 6KB 2024-09-16T00:52:14.014 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.pSXvDYyjNt 2024-09-16T00:52:14.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T00:52:14.091 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T00:52:14.091 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T00:52:14.091 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T00:52:14.091 INFO:teuthology.orchestra.run.smithi078.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T00:52:14.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T00:52:14.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-09-16T00:52:44.256 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-16T00:52:44.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-16T00:52:44.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:53:44.261 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T00:53:53.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-16T00:53:53.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:53:53.621 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-16T00:53:54.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-16T00:53:54.009 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-16T00:53:54.009 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-16T00:53:54.024 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-505.el9.x86_64 2024-09-16T00:53:54.024 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T00:53:54.024 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T00:53:54.024 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T00:53:55.025 DEBUG:teuthology.parallel:result is None 2024-09-16T00:53:55.025 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T00:53:55.035 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T00:53:55.036 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:53:55.038 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:53:55.041 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T00:53:55.059 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T00:53:55.066 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T00:53:55.071 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T00:53:55.072 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:53:55.090 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:53:55.100 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T00:53:55.393 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T00:53:55.400 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T00:53:55.400 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T00:53:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:53:55.417 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T00:53:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:53:55.435 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:53:55.437 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:53:55.460 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T00:53:55.522 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.522 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.538 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.541 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.556 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T00:53:55.561 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T00:53:55.568 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T00:53:55.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:53:55.570 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:53:55.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T00:53:55.600 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T00:53:55.610 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T00:53:55.610 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:53:55.632 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:53:55.641 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T00:53:55.659 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:55.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:55.764 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:55.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:55.881 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:53:55.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:53:55.943 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:55.990 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:56.042 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:56.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:56.170 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:53:56.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:53:56.237 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:56.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T00:53:56.340 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:56.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T00:53:56.468 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:53:56.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T00:53:56.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-09-16T00:53:56.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-09-16T00:53:56.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-09-16T00:53:56.584 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:53:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:53:56.620 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T00:53:57.082 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T00:53:57.093 INFO:teuthology.task.internal:Starting timer... 2024-09-16T00:53:57.093 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T00:53:57.145 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T00:53:57.154 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-16T00:53:57.154 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-09-16T00:53:57.217 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-09-16T00:53:57.219 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-09-16T00:53:57.276 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-09-16T00:53:57.278 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-09-16T00:53:57.324 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-09-16T00:53:57.326 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T00:53:57.326 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-09-16T00:53:57.363 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-09-16T00:53:57.363 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-09-16T00:53:57.383 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-09-16T00:53:57.383 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-09-16T00:53:57.403 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-09-16T00:53:57.404 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-09-16T00:53:57.404 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:53:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:53:57.430 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:53:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:53:57.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T00:53:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T00:53:57.487 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T00:53:57.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:53:57.519 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:53:57.572 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-09-16T00:53:57.605 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T00:53:57.614 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T00:53:57.617 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T00:53:57.736 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T00:53:57.748 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T00:53:57.749 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi148.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T00:58:10.246 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-09-16T00:58:10.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-16T00:58:10.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:58:10.325 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-16T00:58:10.400 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-16T00:58:10.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T00:58:10.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:58:10.476 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-09-16T00:58:10.557 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T00:58:10.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-09-16T00:58:10.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T00:58:10.636 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-09-16T00:58:10.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-09-16T00:58:10.718 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T00:58:10.729 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T00:58:10.730 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:58:10.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:58:10.732 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:58:10.732 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:58:10.735 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T00:58:10.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T00:58:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:58:10.764 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:58:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:58:10.779 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:58:10.789 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T00:58:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-09-16T00:58:10.803 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-09-16T00:58:10.805 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T00:58:10.813 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-09-16T00:58:10.815 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-09-16T00:58:10.827 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:58:10.830 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:58:10.830 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-09-16T00:58:10.841 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:58:10.842 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-09-16T00:58:10.844 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:58:10.856 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T00:58:10.870 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T00:58:10.899 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:58:10.902 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:58:10.902 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-16T00:58:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.903 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.904 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.908 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.933 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-09-16T00:58:10.937 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T00:58:10.937 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-09-16T00:58:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.938 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T00:58:10.938 INFO:teuthology.run_tasks:Running task install... 2024-09-16T00:58:10.946 DEBUG:teuthology.task.install:project ceph 2024-09-16T00:58:10.946 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T00:58:10.946 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T00:58:10.946 INFO:teuthology.task.install:Using flavor: default 2024-09-16T00:58:10.951 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T00:58:10.952 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-16T00:58:10.952 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:58:10.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:58:10.955 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:58:10.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:58:10.956 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T00:58:10.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:58:11.110 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:58:11.111 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:58:11.114 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:58:11.114 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:58:11.117 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T00:58:11.117 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T00:58:11.221 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:58:11.221 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:58:11.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:58:11.233 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:58:11.233 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:58:11.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:58:11.235 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T00:58:11.235 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:58:11.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T00:58:11.249 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:58:11.249 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:58:11.260 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:58:11.261 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:58:11.262 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T00:58:11.262 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T00:58:11.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:58:11.326 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:58:11.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T00:58:11.390 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:58:11.402 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:58:11.403 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T00:58:11.454 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-09-16T00:58:11.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-09-16T00:58:11.466 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2024-09-16T00:58:11.468 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2024-09-16T00:58:11.468 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-09-16T00:58:11.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-09-16T00:58:11.693 INFO:teuthology.orchestra.run.smithi148.stdout:45 files removed 2024-09-16T00:58:11.694 INFO:teuthology.orchestra.run.smithi078.stdout:45 files removed 2024-09-16T00:58:11.696 INFO:teuthology.orchestra.run.smithi193.stdout:45 files removed 2024-09-16T00:58:11.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:58:11.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:58:11.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T00:58:12.266 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 311 kB/s | 83 kB 00:00 2024-09-16T00:58:12.275 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 313 kB/s | 83 kB 00:00 2024-09-16T00:58:12.283 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 299 kB/s | 83 kB 00:00 2024-09-16T00:58:12.542 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 50 kB/s | 12 kB 00:00 2024-09-16T00:58:12.567 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-16T00:58:12.575 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-16T00:58:12.809 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T00:58:12.842 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T00:58:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T00:58:13.368 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-16T00:58:13.560 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T00:58:13.634 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-16T00:58:15.178 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-16T00:58:15.377 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-16T00:58:15.819 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-16T00:58:19.512 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-16T00:58:19.686 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-16T00:58:20.562 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.5 MB 00:00 2024-09-16T00:58:21.054 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-16T00:58:21.362 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 19 kB 00:00 2024-09-16T00:58:21.942 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T00:58:22.446 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 19 kB 00:00 2024-09-16T00:58:22.475 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-16T00:58:23.317 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T00:58:27.398 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-16T00:58:27.906 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-16T00:58:28.872 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T00:58:29.734 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:29.734 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:29.746 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:58:29.747 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:58:29.747 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:58:29.748 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:58:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:58:29.752 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:58:29.753 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:58:29.816 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:58:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:58:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:58:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:58:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:58:29.840 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:58:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:58:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:58:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:58:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:58:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:58:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:58:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:58:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:58:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout:====================================================================================== 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout:Install 148 Packages 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 8 Packages 2024-09-16T00:58:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:58:29.851 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 226 M 2024-09-16T00:58:29.851 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-09-16T00:58:30.274 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:30.275 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:30.286 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:58:30.287 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:58:30.288 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:58:30.288 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:58:30.290 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:58:30.292 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:58:30.293 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:58:30.356 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:58:30.377 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:58:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:58:30.379 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:58:30.380 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:58:30.381 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:58:30.382 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:58:30.383 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:58:30.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:58:30.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:58:30.386 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:58:30.387 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:58:30.388 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:58:30.389 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-09-16T00:58:30.390 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-09-16T00:58:30.390 INFO:teuthology.orchestra.run.smithi193.stdout:Install 148 Packages 2024-09-16T00:58:30.390 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 8 Packages 2024-09-16T00:58:30.390 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:58:30.390 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 226 M 2024-09-16T00:58:30.391 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-09-16T00:58:30.633 INFO:teuthology.orchestra.run.smithi078.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-16T00:58:30.707 INFO:teuthology.orchestra.run.smithi193.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-16T00:58:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.9 MB/s | 896 kB 00:00 2024-09-16T00:58:30.942 INFO:teuthology.orchestra.run.smithi078.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.7 MB/s | 145 kB 00:00 2024-09-16T00:58:31.092 INFO:teuthology.orchestra.run.smithi078.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 9.3 MB/s | 5.4 MB 00:00 2024-09-16T00:58:31.174 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:31.175 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T00:58:31.186 INFO:teuthology.orchestra.run.smithi148.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T00:58:31.187 INFO:teuthology.orchestra.run.smithi148.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T00:58:31.188 INFO:teuthology.orchestra.run.smithi148.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T00:58:31.188 INFO:teuthology.orchestra.run.smithi148.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T00:58:31.190 INFO:teuthology.orchestra.run.smithi148.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T00:58:31.193 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T00:58:31.193 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T00:58:31.241 INFO:teuthology.orchestra.run.smithi193.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.6 MB/s | 896 kB 00:00 2024-09-16T00:58:31.250 INFO:teuthology.orchestra.run.smithi078.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 7.7 MB/s | 2.4 MB 00:00 2024-09-16T00:58:31.255 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T00:58:31.268 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T00:58:31.269 INFO:teuthology.orchestra.run.smithi148.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T00:58:31.270 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T00:58:31.271 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T00:58:31.271 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T00:58:31.271 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T00:58:31.272 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T00:58:31.273 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T00:58:31.274 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T00:58:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T00:58:31.276 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T00:58:31.277 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T00:58:31.278 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T00:58:31.279 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T00:58:31.280 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T00:58:31.281 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2024-09-16T00:58:31.282 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout:Install 148 Packages 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 8 Packages 2024-09-16T00:58:31.283 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:58:31.284 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 226 M 2024-09-16T00:58:31.284 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T00:58:31.284 INFO:teuthology.orchestra.run.smithi078.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 7.9 MB/s | 1.5 MB 00:00 2024-09-16T00:58:31.291 INFO:teuthology.orchestra.run.smithi193.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 2.8 MB/s | 145 kB 00:00 2024-09-16T00:58:31.660 INFO:teuthology.orchestra.run.smithi078.stdout:(7/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 18 MB/s | 21 MB 00:01 2024-09-16T00:58:31.735 INFO:teuthology.orchestra.run.smithi078.stdout:(8/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 9.7 MB/s | 4.7 MB 00:00 2024-09-16T00:58:31.793 INFO:teuthology.orchestra.run.smithi078.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 433 kB/s | 25 kB 00:00 2024-09-16T00:58:32.007 INFO:teuthology.orchestra.run.smithi148.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2024-09-16T00:58:32.199 INFO:teuthology.orchestra.run.smithi193.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-09-16T00:58:32.227 INFO:teuthology.orchestra.run.smithi078.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 19 MB/s | 11 MB 00:00 2024-09-16T00:58:32.274 INFO:teuthology.orchestra.run.smithi148.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.3 MB/s | 896 kB 00:00 2024-09-16T00:58:32.324 INFO:teuthology.orchestra.run.smithi148.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 2.9 MB/s | 145 kB 00:00 2024-09-16T00:58:32.341 INFO:teuthology.orchestra.run.smithi193.stdout:(5/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 3.1 MB/s | 5.4 MB 00:01 2024-09-16T00:58:32.402 INFO:teuthology.orchestra.run.smithi078.stdout:(11/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 15 MB/s | 17 MB 00:01 2024-09-16T00:58:32.427 INFO:teuthology.orchestra.run.smithi078.stdout:(12/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 1.1 MB/s | 228 kB 00:00 2024-09-16T00:58:32.449 INFO:teuthology.orchestra.run.smithi148.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 19 MB/s | 2.4 MB 00:00 2024-09-16T00:58:32.453 INFO:teuthology.orchestra.run.smithi078.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 543 kB/s | 27 kB 00:00 2024-09-16T00:58:32.478 INFO:teuthology.orchestra.run.smithi078.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 634 kB/s | 32 kB 00:00 2024-09-16T00:58:32.541 INFO:teuthology.orchestra.run.smithi148.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 17 MB/s | 1.5 MB 00:00 2024-09-16T00:58:32.570 INFO:teuthology.orchestra.run.smithi078.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.7 MB/s | 163 kB 00:00 2024-09-16T00:58:32.583 INFO:teuthology.orchestra.run.smithi193.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-09-16T00:58:32.595 INFO:teuthology.orchestra.run.smithi078.stdout:(16/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 5.2 MB/s | 751 kB 00:00 2024-09-16T00:58:32.654 INFO:teuthology.orchestra.run.smithi078.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 1.5 MB/s | 127 kB 00:00 2024-09-16T00:58:32.704 INFO:teuthology.orchestra.run.smithi078.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 4.5 MB/s | 506 kB 00:00 2024-09-16T00:58:32.767 INFO:teuthology.orchestra.run.smithi148.stdout:(6/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 21 MB/s | 4.7 MB 00:00 2024-09-16T00:58:32.787 INFO:teuthology.orchestra.run.smithi078.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 543 kB/s | 45 kB 00:00 2024-09-16T00:58:33.367 INFO:teuthology.orchestra.run.smithi148.stdout:(7/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 3.3 MB/s | 5.4 MB 00:01 2024-09-16T00:58:33.505 INFO:teuthology.orchestra.run.smithi078.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 190 kB/s | 136 kB 00:00 2024-09-16T00:58:33.555 INFO:teuthology.orchestra.run.smithi078.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 3.2 MB/s | 161 kB 00:00 2024-09-16T00:58:33.583 INFO:teuthology.orchestra.run.smithi193.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 3.8 MB/s | 4.7 MB 00:01 2024-09-16T00:58:33.622 INFO:teuthology.orchestra.run.smithi078.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.7 MB/s | 320 kB 00:00 2024-09-16T00:58:33.642 INFO:teuthology.orchestra.run.smithi148.stdout:(8/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 19 MB/s | 17 MB 00:00 2024-09-16T00:58:33.680 INFO:teuthology.orchestra.run.smithi078.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 5.0 MB/s | 300 kB 00:00 2024-09-16T00:58:33.709 INFO:teuthology.orchestra.run.smithi148.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 380 kB/s | 25 kB 00:00 2024-09-16T00:58:33.781 INFO:teuthology.orchestra.run.smithi078.stdout:(24/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 4.7 MB/s | 5.3 MB 00:01 2024-09-16T00:58:33.806 INFO:teuthology.orchestra.run.smithi078.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 793 kB/s | 99 kB 00:00 2024-09-16T00:58:33.865 INFO:teuthology.orchestra.run.smithi078.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.0 MB/s | 85 kB 00:00 2024-09-16T00:58:33.982 INFO:teuthology.orchestra.run.smithi078.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 1.4 MB/s | 170 kB 00:00 2024-09-16T00:58:34.082 INFO:teuthology.orchestra.run.smithi078.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 318 kB/s | 32 kB 00:00 2024-09-16T00:58:34.157 INFO:teuthology.orchestra.run.smithi078.stdout:(29/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 8.8 MB/s | 3.1 MB 00:00 2024-09-16T00:58:34.182 INFO:teuthology.orchestra.run.smithi078.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.4 MB/s | 146 kB 00:00 2024-09-16T00:58:34.683 INFO:teuthology.orchestra.run.smithi078.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 6.8 MB/s | 3.6 MB 00:00 2024-09-16T00:58:34.817 INFO:teuthology.orchestra.run.smithi078.stdout:(32/156): ceph-mgr-diskprediction-local-19.1.1- 12 MB/s | 7.4 MB 00:00 2024-09-16T00:58:34.842 INFO:teuthology.orchestra.run.smithi078.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 1.5 MB/s | 249 kB 00:00 2024-09-16T00:58:34.867 INFO:teuthology.orchestra.run.smithi078.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 984 kB/s | 49 kB 00:00 2024-09-16T00:58:34.876 INFO:teuthology.orchestra.run.smithi148.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 7.0 MB/s | 11 MB 00:01 2024-09-16T00:58:34.884 INFO:teuthology.orchestra.run.smithi078.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 390 kB/s | 16 kB 00:00 2024-09-16T00:58:34.935 INFO:teuthology.orchestra.run.smithi078.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.2 MB/s | 285 kB 00:00 2024-09-16T00:58:34.943 INFO:teuthology.orchestra.run.smithi148.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.3 MB/s | 228 kB 00:00 2024-09-16T00:58:35.001 INFO:teuthology.orchestra.run.smithi078.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 6.4 MB/s | 765 kB 00:00 2024-09-16T00:58:35.010 INFO:teuthology.orchestra.run.smithi148.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 407 kB/s | 27 kB 00:00 2024-09-16T00:58:35.026 INFO:teuthology.orchestra.run.smithi078.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 199 kB/s | 18 kB 00:00 2024-09-16T00:58:35.068 INFO:teuthology.orchestra.run.smithi148.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 543 kB/s | 32 kB 00:00 2024-09-16T00:58:35.177 INFO:teuthology.orchestra.run.smithi148.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 6.8 MB/s | 751 kB 00:00 2024-09-16T00:58:35.252 INFO:teuthology.orchestra.run.smithi148.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.1 MB/s | 163 kB 00:00 2024-09-16T00:58:35.469 INFO:teuthology.orchestra.run.smithi148.stdout:(16/156): ceph-common-19.1.1-326.g8921c8ef.el9. 5.6 MB/s | 21 MB 00:03 2024-09-16T00:58:35.477 INFO:teuthology.orchestra.run.smithi078.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 168 kB/s | 80 kB 00:00 2024-09-16T00:58:35.494 INFO:teuthology.orchestra.run.smithi148.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 524 kB/s | 127 kB 00:00 2024-09-16T00:58:35.553 INFO:teuthology.orchestra.run.smithi148.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 5.9 MB/s | 506 kB 00:00 2024-09-16T00:58:35.612 INFO:teuthology.orchestra.run.smithi148.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 776 kB/s | 45 kB 00:00 2024-09-16T00:58:35.679 INFO:teuthology.orchestra.run.smithi148.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.0 MB/s | 136 kB 00:00 2024-09-16T00:58:35.685 INFO:teuthology.orchestra.run.smithi078.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 918 kB/s | 191 kB 00:00 2024-09-16T00:58:35.737 INFO:teuthology.orchestra.run.smithi148.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.7 MB/s | 161 kB 00:00 2024-09-16T00:58:35.744 INFO:teuthology.orchestra.run.smithi078.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-16T00:58:35.803 INFO:teuthology.orchestra.run.smithi078.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-16T00:58:35.812 INFO:teuthology.orchestra.run.smithi148.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-16T00:58:35.880 INFO:teuthology.orchestra.run.smithi148.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 4.4 MB/s | 300 kB 00:00 2024-09-16T00:58:35.928 INFO:teuthology.orchestra.run.smithi078.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 852 kB/s | 106 kB 00:00 2024-09-16T00:58:35.947 INFO:teuthology.orchestra.run.smithi148.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.5 MB/s | 99 kB 00:00 2024-09-16T00:58:35.962 INFO:teuthology.orchestra.run.smithi078.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-09-16T00:58:36.022 INFO:teuthology.orchestra.run.smithi148.stdout:(25/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 10 MB/s | 5.3 MB 00:00 2024-09-16T00:58:36.046 INFO:teuthology.orchestra.run.smithi078.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-16T00:58:36.047 INFO:teuthology.orchestra.run.smithi148.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 851 kB/s | 85 kB 00:00 2024-09-16T00:58:36.071 INFO:teuthology.orchestra.run.smithi078.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-16T00:58:36.114 INFO:teuthology.orchestra.run.smithi148.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 170 kB 00:00 2024-09-16T00:58:36.163 INFO:teuthology.orchestra.run.smithi078.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-09-16T00:58:36.181 INFO:teuthology.orchestra.run.smithi148.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 479 kB/s | 32 kB 00:00 2024-09-16T00:58:36.221 INFO:teuthology.orchestra.run.smithi078.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 4.5 MB/s | 689 kB 00:00 2024-09-16T00:58:36.248 INFO:teuthology.orchestra.run.smithi148.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.1 MB/s | 146 kB 00:00 2024-09-16T00:58:36.263 INFO:teuthology.orchestra.run.smithi078.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-09-16T00:58:36.306 INFO:teuthology.orchestra.run.smithi148.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 3.1 MB 00:00 2024-09-16T00:58:36.350 INFO:teuthology.orchestra.run.smithi193.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 3.7 MB/s | 21 MB 00:05 2024-09-16T00:58:36.417 INFO:teuthology.orchestra.run.smithi193.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 379 kB/s | 25 kB 00:00 2024-09-16T00:58:36.680 INFO:teuthology.orchestra.run.smithi078.stdout:(50/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 10 MB/s | 49 MB 00:04 2024-09-16T00:58:36.706 INFO:teuthology.orchestra.run.smithi078.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 49 kB/s | 22 kB 00:00 2024-09-16T00:58:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 1.6 MB/s | 943 kB 00:00 2024-09-16T00:58:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-09-16T00:58:36.801 INFO:teuthology.orchestra.run.smithi193.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 3.3 MB/s | 11 MB 00:03 2024-09-16T00:58:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 440 kB/s | 18 kB 00:00 2024-09-16T00:58:36.849 INFO:teuthology.orchestra.run.smithi078.stdout:(55/156): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-09-16T00:58:36.874 INFO:teuthology.orchestra.run.smithi078.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-09-16T00:58:36.884 INFO:teuthology.orchestra.run.smithi193.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 2.7 MB/s | 228 kB 00:00 2024-09-16T00:58:36.899 INFO:teuthology.orchestra.run.smithi078.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-16T00:58:36.924 INFO:teuthology.orchestra.run.smithi078.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-09-16T00:58:36.942 INFO:teuthology.orchestra.run.smithi193.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 464 kB/s | 27 kB 00:00 2024-09-16T00:58:36.983 INFO:teuthology.orchestra.run.smithi078.stdout:(59/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-09-16T00:58:37.001 INFO:teuthology.orchestra.run.smithi193.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 543 kB/s | 32 kB 00:00 2024-09-16T00:58:37.017 INFO:teuthology.orchestra.run.smithi078.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-16T00:58:37.042 INFO:teuthology.orchestra.run.smithi078.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 418 kB/s | 24 kB 00:00 2024-09-16T00:58:37.067 INFO:teuthology.orchestra.run.smithi078.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-16T00:58:37.074 INFO:teuthology.orchestra.run.smithi148.stdout:(31/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 14 MB/s | 49 MB 00:03 2024-09-16T00:58:37.092 INFO:teuthology.orchestra.run.smithi078.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-16T00:58:37.117 INFO:teuthology.orchestra.run.smithi078.stdout:(64/156): libtool-2.4.6-46.el9.x86_64.rpm 2.9 MB/s | 581 kB 00:00 2024-09-16T00:58:37.141 INFO:teuthology.orchestra.run.smithi148.stdout:(32/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 4.0 MB/s | 3.6 MB 00:00 2024-09-16T00:58:37.143 INFO:teuthology.orchestra.run.smithi078.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 767 kB/s | 38 kB 00:00 2024-09-16T00:58:37.166 INFO:teuthology.orchestra.run.smithi148.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 2.7 MB/s | 249 kB 00:00 2024-09-16T00:58:37.168 INFO:teuthology.orchestra.run.smithi078.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-16T00:58:37.185 INFO:teuthology.orchestra.run.smithi193.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 4.0 MB/s | 751 kB 00:00 2024-09-16T00:58:37.260 INFO:teuthology.orchestra.run.smithi193.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.1 MB/s | 163 kB 00:00 2024-09-16T00:58:37.266 INFO:teuthology.orchestra.run.smithi148.stdout:(34/156): ceph-mgr-diskprediction-local-19.1.1- 7.7 MB/s | 7.4 MB 00:00 2024-09-16T00:58:37.277 INFO:teuthology.orchestra.run.smithi078.stdout:(67/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-09-16T00:58:37.292 INFO:teuthology.orchestra.run.smithi148.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 131 kB/s | 16 kB 00:00 2024-09-16T00:58:37.317 INFO:teuthology.orchestra.run.smithi148.stdout:(36/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 281 kB/s | 49 kB 00:00 2024-09-16T00:58:37.319 INFO:teuthology.orchestra.run.smithi078.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-16T00:58:37.342 INFO:teuthology.orchestra.run.smithi148.stdout:(37/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 3.7 MB/s | 285 kB 00:00 2024-09-16T00:58:37.352 INFO:teuthology.orchestra.run.smithi078.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-09-16T00:58:37.380 INFO:teuthology.orchestra.run.smithi078.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-16T00:58:37.393 INFO:teuthology.orchestra.run.smithi148.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 241 kB/s | 18 kB 00:00 2024-09-16T00:58:37.426 INFO:teuthology.orchestra.run.smithi148.stdout:(39/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 5.6 MB/s | 765 kB 00:00 2024-09-16T00:58:37.427 INFO:teuthology.orchestra.run.smithi193.stdout:(16/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 3.5 MB/s | 17 MB 00:04 2024-09-16T00:58:37.445 INFO:teuthology.orchestra.run.smithi078.stdout:(71/156): openblas-openmp-0.3.26-2.el9.x86_64.r 15 MB/s | 4.9 MB 00:00 2024-09-16T00:58:37.452 INFO:teuthology.orchestra.run.smithi193.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 661 kB/s | 127 kB 00:00 2024-09-16T00:58:37.478 INFO:teuthology.orchestra.run.smithi078.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-09-16T00:58:37.512 INFO:teuthology.orchestra.run.smithi078.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-16T00:58:37.535 INFO:teuthology.orchestra.run.smithi148.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 416 kB/s | 80 kB 00:00 2024-09-16T00:58:37.554 INFO:teuthology.orchestra.run.smithi078.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-16T00:58:37.603 INFO:teuthology.orchestra.run.smithi193.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 2.8 MB/s | 506 kB 00:00 2024-09-16T00:58:37.646 INFO:teuthology.orchestra.run.smithi078.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-09-16T00:58:37.652 INFO:teuthology.orchestra.run.smithi148.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-16T00:58:37.671 INFO:teuthology.orchestra.run.smithi078.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-16T00:58:37.677 INFO:teuthology.orchestra.run.smithi148.stdout:(42/156): libquadmath-11.5.0-2.el9.x86_64.rpm 763 kB/s | 191 kB 00:00 2024-09-16T00:58:37.696 INFO:teuthology.orchestra.run.smithi078.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-16T00:58:37.711 INFO:teuthology.orchestra.run.smithi148.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-16T00:58:37.720 INFO:teuthology.orchestra.run.smithi193.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 387 kB/s | 45 kB 00:00 2024-09-16T00:58:37.721 INFO:teuthology.orchestra.run.smithi078.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-09-16T00:58:37.736 INFO:teuthology.orchestra.run.smithi148.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-16T00:58:37.747 INFO:teuthology.orchestra.run.smithi078.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-09-16T00:58:37.761 INFO:teuthology.orchestra.run.smithi148.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-16T00:58:37.772 INFO:teuthology.orchestra.run.smithi078.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-16T00:58:37.788 INFO:teuthology.orchestra.run.smithi148.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-16T00:58:37.798 INFO:teuthology.orchestra.run.smithi078.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 690 kB/s | 35 kB 00:00 2024-09-16T00:58:37.803 INFO:teuthology.orchestra.run.smithi193.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.6 MB/s | 136 kB 00:00 2024-09-16T00:58:37.812 INFO:teuthology.orchestra.run.smithi148.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-16T00:58:37.879 INFO:teuthology.orchestra.run.smithi193.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.1 MB/s | 161 kB 00:00 2024-09-16T00:58:37.898 INFO:teuthology.orchestra.run.smithi078.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-16T00:58:37.923 INFO:teuthology.orchestra.run.smithi078.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 250 kB 00:00 2024-09-16T00:58:37.948 INFO:teuthology.orchestra.run.smithi078.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-16T00:58:37.982 INFO:teuthology.orchestra.run.smithi078.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-16T00:58:38.004 INFO:teuthology.orchestra.run.smithi148.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 3.5 MB/s | 1.2 MB 00:00 2024-09-16T00:58:38.007 INFO:teuthology.orchestra.run.smithi078.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 720 kB/s | 42 kB 00:00 2024-09-16T00:58:38.041 INFO:teuthology.orchestra.run.smithi078.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-16T00:58:38.045 INFO:teuthology.orchestra.run.smithi193.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 1.9 MB/s | 320 kB 00:00 2024-09-16T00:58:38.066 INFO:teuthology.orchestra.run.smithi078.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-16T00:58:38.100 INFO:teuthology.orchestra.run.smithi078.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-09-16T00:58:38.125 INFO:teuthology.orchestra.run.smithi078.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-09-16T00:58:38.150 INFO:teuthology.orchestra.run.smithi078.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-16T00:58:38.221 INFO:teuthology.orchestra.run.smithi193.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 300 kB 00:00 2024-09-16T00:58:38.226 INFO:teuthology.orchestra.run.smithi078.stdout:(92/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-16T00:58:38.251 INFO:teuthology.orchestra.run.smithi078.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 414 kB/s | 42 kB 00:00 2024-09-16T00:58:38.276 INFO:teuthology.orchestra.run.smithi078.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 881 kB/s | 44 kB 00:00 2024-09-16T00:58:38.302 INFO:teuthology.orchestra.run.smithi078.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-16T00:58:38.312 INFO:teuthology.orchestra.run.smithi148.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 347 kB/s | 107 kB 00:00 2024-09-16T00:58:38.327 INFO:teuthology.orchestra.run.smithi078.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-09-16T00:58:38.352 INFO:teuthology.orchestra.run.smithi078.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-09-16T00:58:38.371 INFO:teuthology.orchestra.run.smithi148.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-09-16T00:58:38.386 INFO:teuthology.orchestra.run.smithi078.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-16T00:58:38.411 INFO:teuthology.orchestra.run.smithi078.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-09-16T00:58:38.421 INFO:teuthology.orchestra.run.smithi148.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 689 kB 00:00 2024-09-16T00:58:38.437 INFO:teuthology.orchestra.run.smithi078.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-16T00:58:38.462 INFO:teuthology.orchestra.run.smithi078.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T00:58:38.478 INFO:teuthology.orchestra.run.smithi078.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-09-16T00:58:38.488 INFO:teuthology.orchestra.run.smithi148.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 492 kB/s | 33 kB 00:00 2024-09-16T00:58:38.605 INFO:teuthology.orchestra.run.smithi148.stdout:(53/156): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-09-16T00:58:38.672 INFO:teuthology.orchestra.run.smithi148.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 273 kB/s | 18 kB 00:00 2024-09-16T00:58:38.697 INFO:teuthology.orchestra.run.smithi148.stdout:(55/156): flex-2.6.4-9.el9.x86_64.rpm 962 kB/s | 313 kB 00:00 2024-09-16T00:58:38.739 INFO:teuthology.orchestra.run.smithi148.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 166 kB/s | 11 kB 00:00 2024-09-16T00:58:38.761 INFO:teuthology.orchestra.run.smithi078.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-16T00:58:38.763 INFO:teuthology.orchestra.run.smithi193.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 184 kB/s | 99 kB 00:00 2024-09-16T00:58:38.779 INFO:teuthology.orchestra.run.smithi078.stdout:(104/156): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-09-16T00:58:38.815 INFO:teuthology.orchestra.run.smithi148.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-16T00:58:38.829 INFO:teuthology.orchestra.run.smithi078.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 7.9 MB/s | 548 kB 00:00 2024-09-16T00:58:38.830 INFO:teuthology.orchestra.run.smithi193.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.2 MB/s | 85 kB 00:00 2024-09-16T00:58:38.840 INFO:teuthology.orchestra.run.smithi148.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-16T00:58:38.855 INFO:teuthology.orchestra.run.smithi078.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-16T00:58:38.888 INFO:teuthology.orchestra.run.smithi078.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.8 MB/s | 308 kB 00:00 2024-09-16T00:58:38.913 INFO:teuthology.orchestra.run.smithi078.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T00:58:38.939 INFO:teuthology.orchestra.run.smithi078.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T00:58:38.964 INFO:teuthology.orchestra.run.smithi078.stdout:(110/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-09-16T00:58:38.989 INFO:teuthology.orchestra.run.smithi078.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T00:58:39.056 INFO:teuthology.orchestra.run.smithi078.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-16T00:58:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:(113/156): luarocks-3.9.2-5.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-09-16T00:58:39.107 INFO:teuthology.orchestra.run.smithi078.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T00:58:39.115 INFO:teuthology.orchestra.run.smithi148.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-16T00:58:39.132 INFO:teuthology.orchestra.run.smithi078.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-16T00:58:39.158 INFO:teuthology.orchestra.run.smithi078.stdout:(116/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-09-16T00:58:39.183 INFO:teuthology.orchestra.run.smithi078.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-16T00:58:39.190 INFO:teuthology.orchestra.run.smithi148.stdout:(60/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 326 kB/s | 24 kB 00:00 2024-09-16T00:58:39.208 INFO:teuthology.orchestra.run.smithi078.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-16T00:58:39.238 INFO:teuthology.orchestra.run.smithi193.stdout:(26/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 3.0 MB/s | 5.3 MB 00:01 2024-09-16T00:58:39.242 INFO:teuthology.orchestra.run.smithi078.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 8.9 MB/s | 303 kB 00:00 2024-09-16T00:58:39.269 INFO:teuthology.orchestra.run.smithi148.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-09-16T00:58:39.275 INFO:teuthology.orchestra.run.smithi078.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-16T00:58:39.301 INFO:teuthology.orchestra.run.smithi078.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-16T00:58:39.305 INFO:teuthology.orchestra.run.smithi193.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 170 kB 00:00 2024-09-16T00:58:39.359 INFO:teuthology.orchestra.run.smithi078.stdout:(122/156): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-09-16T00:58:39.385 INFO:teuthology.orchestra.run.smithi078.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-09-16T00:58:39.389 INFO:teuthology.orchestra.run.smithi193.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 381 kB/s | 32 kB 00:00 2024-09-16T00:58:39.410 INFO:teuthology.orchestra.run.smithi078.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-09-16T00:58:39.424 INFO:teuthology.orchestra.run.smithi148.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 995 kB/s | 581 kB 00:00 2024-09-16T00:58:39.435 INFO:teuthology.orchestra.run.smithi078.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 718 kB/s | 18 kB 00:00 2024-09-16T00:58:39.449 INFO:teuthology.orchestra.run.smithi148.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-09-16T00:58:39.460 INFO:teuthology.orchestra.run.smithi078.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 464 kB/s | 23 kB 00:00 2024-09-16T00:58:39.473 INFO:teuthology.orchestra.run.smithi193.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.7 MB/s | 146 kB 00:00 2024-09-16T00:58:39.486 INFO:teuthology.orchestra.run.smithi078.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-09-16T00:58:39.511 INFO:teuthology.orchestra.run.smithi078.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-16T00:58:39.516 INFO:teuthology.orchestra.run.smithi148.stdout:(64/156): openblas-0.3.26-2.el9.x86_64.rpm 577 kB/s | 38 kB 00:00 2024-09-16T00:58:39.536 INFO:teuthology.orchestra.run.smithi078.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T00:58:39.570 INFO:teuthology.orchestra.run.smithi078.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-16T00:58:39.592 INFO:teuthology.orchestra.run.smithi148.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-16T00:58:39.603 INFO:teuthology.orchestra.run.smithi078.stdout:(131/156): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-09-16T00:58:39.629 INFO:teuthology.orchestra.run.smithi078.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 984 kB/s | 58 kB 00:00 2024-09-16T00:58:39.654 INFO:teuthology.orchestra.run.smithi078.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-09-16T00:58:39.688 INFO:teuthology.orchestra.run.smithi078.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.6 MB/s | 90 kB 00:00 2024-09-16T00:58:39.692 INFO:teuthology.orchestra.run.smithi148.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-09-16T00:58:39.714 INFO:teuthology.orchestra.run.smithi078.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-16T00:58:39.739 INFO:teuthology.orchestra.run.smithi078.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-16T00:58:39.764 INFO:teuthology.orchestra.run.smithi078.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-09-16T00:58:39.806 INFO:teuthology.orchestra.run.smithi078.stdout:(138/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-09-16T00:58:39.831 INFO:teuthology.orchestra.run.smithi078.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-09-16T00:58:39.857 INFO:teuthology.orchestra.run.smithi078.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-09-16T00:58:39.882 INFO:teuthology.orchestra.run.smithi078.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-16T00:58:39.907 INFO:teuthology.orchestra.run.smithi078.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 988 kB/s | 74 kB 00:00 2024-09-16T00:58:39.933 INFO:teuthology.orchestra.run.smithi078.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T00:58:39.956 INFO:teuthology.orchestra.run.smithi193.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 2.8 MB/s | 3.1 MB 00:01 2024-09-16T00:58:39.966 INFO:teuthology.orchestra.run.smithi078.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-16T00:58:39.991 INFO:teuthology.orchestra.run.smithi078.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 340 kB/s | 20 kB 00:00 2024-09-16T00:58:40.017 INFO:teuthology.orchestra.run.smithi078.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T00:58:40.059 INFO:teuthology.orchestra.run.smithi078.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-09-16T00:58:40.109 INFO:teuthology.orchestra.run.smithi078.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T00:58:40.209 INFO:teuthology.orchestra.run.smithi148.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-09-16T00:58:40.526 INFO:teuthology.orchestra.run.smithi148.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 2.8 MB/s | 910 kB 00:00 2024-09-16T00:58:40.593 INFO:teuthology.orchestra.run.smithi148.stdout:(69/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-16T00:58:40.851 INFO:teuthology.orchestra.run.smithi078.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 3.1 MB/s | 2.3 MB 00:00 2024-09-16T00:58:40.907 INFO:teuthology.orchestra.run.smithi193.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 2.5 MB/s | 3.6 MB 00:01 2024-09-16T00:58:40.993 INFO:teuthology.orchestra.run.smithi078.stdout:(150/156): NetworkManager-libnm-1.49.90-1.el9.x 13 MB/s | 1.8 MB 00:00 2024-09-16T00:58:41.065 INFO:teuthology.orchestra.run.smithi193.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 1.5 MB/s | 249 kB 00:00 2024-09-16T00:58:41.101 INFO:teuthology.orchestra.run.smithi148.stdout:(70/156): openblas-openmp-0.3.26-2.el9.x86_64.r 3.1 MB/s | 4.9 MB 00:01 2024-09-16T00:58:41.110 INFO:teuthology.orchestra.run.smithi078.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 350 kB/s | 41 kB 00:00 2024-09-16T00:58:41.165 INFO:teuthology.orchestra.run.smithi193.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 492 kB/s | 49 kB 00:00 2024-09-16T00:58:41.227 INFO:teuthology.orchestra.run.smithi078.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 2.1 MB/s | 249 kB 00:00 2024-09-16T00:58:41.274 INFO:teuthology.orchestra.run.smithi193.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 152 kB/s | 16 kB 00:00 2024-09-16T00:58:41.360 INFO:teuthology.orchestra.run.smithi078.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-09-16T00:58:41.382 INFO:teuthology.orchestra.run.smithi193.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 2.6 MB/s | 285 kB 00:00 2024-09-16T00:58:41.418 INFO:teuthology.orchestra.run.smithi148.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-16T00:58:41.477 INFO:teuthology.orchestra.run.smithi078.stdout:(154/156): librados2-19.1.1-326.g8921c8ef.el9.x 2.3 MB/s | 3.4 MB 00:01 2024-09-16T00:58:41.494 INFO:teuthology.orchestra.run.smithi148.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-09-16T00:58:41.527 INFO:teuthology.orchestra.run.smithi078.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-09-16T00:58:41.550 INFO:teuthology.orchestra.run.smithi193.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 4.5 MB/s | 765 kB 00:00 2024-09-16T00:58:41.592 INFO:teuthology.orchestra.run.smithi193.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 441 kB/s | 18 kB 00:00 2024-09-16T00:58:41.725 INFO:teuthology.orchestra.run.smithi193.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 598 kB/s | 80 kB 00:00 2024-09-16T00:58:41.819 INFO:teuthology.orchestra.run.smithi148.stdout:(73/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-09-16T00:58:41.926 INFO:teuthology.orchestra.run.smithi193.stdout:(39/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-16T00:58:41.944 INFO:teuthology.orchestra.run.smithi148.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 491 kB/s | 61 kB 00:00 2024-09-16T00:58:41.976 INFO:teuthology.orchestra.run.smithi193.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T00:58:42.043 INFO:teuthology.orchestra.run.smithi193.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-09-16T00:58:42.061 INFO:teuthology.orchestra.run.smithi148.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-16T00:58:42.149 INFO:teuthology.orchestra.run.smithi148.stdout:(76/156): postgresql-server-13.16-1.el9.x86_64. 3.7 MB/s | 5.8 MB 00:01 2024-09-16T00:58:42.186 INFO:teuthology.orchestra.run.smithi078.stdout:(156/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 1.5 MB/s | 3.2 MB 00:02 2024-09-16T00:58:42.193 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:58:42.193 INFO:teuthology.orchestra.run.smithi078.stdout:Total 18 MB/s | 226 MB 00:12 2024-09-16T00:58:42.218 INFO:teuthology.orchestra.run.smithi193.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 7.0 MB/s | 1.2 MB 00:00 2024-09-16T00:58:42.220 INFO:teuthology.orchestra.run.smithi148.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-09-16T00:58:42.245 INFO:teuthology.orchestra.run.smithi148.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 266 kB/s | 26 kB 00:00 2024-09-16T00:58:42.269 INFO:teuthology.orchestra.run.smithi193.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-16T00:58:42.287 INFO:teuthology.orchestra.run.smithi148.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-09-16T00:58:42.313 INFO:teuthology.orchestra.run.smithi148.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-16T00:58:42.327 INFO:teuthology.orchestra.run.smithi193.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-16T00:58:42.354 INFO:teuthology.orchestra.run.smithi148.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-09-16T00:58:42.411 INFO:teuthology.orchestra.run.smithi193.stdout:(45/156): ceph-mgr-diskprediction-local-19.1.1- 3.0 MB/s | 7.4 MB 00:02 2024-09-16T00:58:42.436 INFO:teuthology.orchestra.run.smithi193.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-16T00:58:42.465 INFO:teuthology.orchestra.run.smithi148.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-09-16T00:58:42.578 INFO:teuthology.orchestra.run.smithi193.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 4.7 MB/s | 689 kB 00:00 2024-09-16T00:58:42.580 INFO:teuthology.orchestra.run.smithi148.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-09-16T00:58:42.604 INFO:teuthology.orchestra.run.smithi193.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.1 MB/s | 215 kB 00:00 2024-09-16T00:58:42.647 INFO:teuthology.orchestra.run.smithi148.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-09-16T00:58:42.654 INFO:teuthology.orchestra.run.smithi193.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-09-16T00:58:42.688 INFO:teuthology.orchestra.run.smithi193.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 653 kB/s | 22 kB 00:00 2024-09-16T00:58:42.713 INFO:teuthology.orchestra.run.smithi193.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 982 kB/s | 107 kB 00:00 2024-09-16T00:58:42.714 INFO:teuthology.orchestra.run.smithi148.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 635 kB/s | 42 kB 00:00 2024-09-16T00:58:42.738 INFO:teuthology.orchestra.run.smithi193.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-09-16T00:58:42.780 INFO:teuthology.orchestra.run.smithi193.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-09-16T00:58:42.826 INFO:teuthology.orchestra.run.smithi193.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 398 kB/s | 18 kB 00:00 2024-09-16T00:58:42.839 INFO:teuthology.orchestra.run.smithi148.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-09-16T00:58:42.877 INFO:teuthology.orchestra.run.smithi193.stdout:(55/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-09-16T00:58:42.902 INFO:teuthology.orchestra.run.smithi193.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-09-16T00:58:42.927 INFO:teuthology.orchestra.run.smithi193.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-16T00:58:42.948 INFO:teuthology.orchestra.run.smithi148.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T00:58:42.961 INFO:teuthology.orchestra.run.smithi193.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-09-16T00:58:42.995 INFO:teuthology.orchestra.run.smithi193.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 662 kB 00:00 2024-09-16T00:58:43.037 INFO:teuthology.orchestra.run.smithi193.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 7.5 MB/s | 581 kB 00:00 2024-09-16T00:58:43.062 INFO:teuthology.orchestra.run.smithi193.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 365 kB/s | 24 kB 00:00 2024-09-16T00:58:43.064 INFO:teuthology.orchestra.run.smithi148.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-16T00:58:43.087 INFO:teuthology.orchestra.run.smithi193.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-09-16T00:58:43.113 INFO:teuthology.orchestra.run.smithi193.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-16T00:58:43.138 INFO:teuthology.orchestra.run.smithi193.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T00:58:43.155 INFO:teuthology.orchestra.run.smithi193.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 922 kB/s | 38 kB 00:00 2024-09-16T00:58:43.173 INFO:teuthology.orchestra.run.smithi148.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 470 kB/s | 51 kB 00:00 2024-09-16T00:58:43.305 INFO:teuthology.orchestra.run.smithi193.stdout:(66/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-09-16T00:58:43.323 INFO:teuthology.orchestra.run.smithi148.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 358 kB/s | 54 kB 00:00 2024-09-16T00:58:43.381 INFO:teuthology.orchestra.run.smithi193.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 20 MB/s | 4.9 MB 00:00 2024-09-16T00:58:43.422 INFO:teuthology.orchestra.run.smithi193.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T00:58:43.431 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-09-16T00:58:43.464 INFO:teuthology.orchestra.run.smithi193.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 11 MB/s | 910 kB 00:00 2024-09-16T00:58:43.490 INFO:teuthology.orchestra.run.smithi193.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-16T00:58:43.524 INFO:teuthology.orchestra.run.smithi148.stdout:(91/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-09-16T00:58:43.565 INFO:teuthology.orchestra.run.smithi193.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-16T00:58:43.580 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-09-16T00:58:43.580 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-09-16T00:58:43.590 INFO:teuthology.orchestra.run.smithi148.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 626 kB/s | 42 kB 00:00 2024-09-16T00:58:43.640 INFO:teuthology.orchestra.run.smithi193.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-09-16T00:58:43.657 INFO:teuthology.orchestra.run.smithi148.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 662 kB/s | 44 kB 00:00 2024-09-16T00:58:43.694 INFO:teuthology.orchestra.run.smithi193.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 873 kB/s | 87 kB 00:00 2024-09-16T00:58:43.716 INFO:teuthology.orchestra.run.smithi193.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-16T00:58:43.741 INFO:teuthology.orchestra.run.smithi148.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T00:58:43.791 INFO:teuthology.orchestra.run.smithi193.stdout:(75/156): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-09-16T00:58:43.816 INFO:teuthology.orchestra.run.smithi193.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-16T00:58:43.833 INFO:teuthology.orchestra.run.smithi148.stdout:(95/156): python3-babel-2.9.1-2.el9.noarch.rpm 2.5 MB/s | 6.0 MB 00:02 2024-09-16T00:58:43.841 INFO:teuthology.orchestra.run.smithi193.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-09-16T00:58:43.858 INFO:teuthology.orchestra.run.smithi148.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 307 kB/s | 36 kB 00:00 2024-09-16T00:58:43.867 INFO:teuthology.orchestra.run.smithi193.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-09-16T00:58:43.892 INFO:teuthology.orchestra.run.smithi193.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-09-16T00:58:43.909 INFO:teuthology.orchestra.run.smithi148.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-16T00:58:43.909 INFO:teuthology.orchestra.run.smithi193.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-16T00:58:43.943 INFO:teuthology.orchestra.run.smithi193.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-09-16T00:58:43.993 INFO:teuthology.orchestra.run.smithi193.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-09-16T00:58:44.026 INFO:teuthology.orchestra.run.smithi148.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-16T00:58:44.051 INFO:teuthology.orchestra.run.smithi148.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 407 kB/s | 58 kB 00:00 2024-09-16T00:58:44.085 INFO:teuthology.orchestra.run.smithi193.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-09-16T00:58:44.093 INFO:teuthology.orchestra.run.smithi148.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.4 MB/s | 96 kB 00:00 2024-09-16T00:58:44.110 INFO:teuthology.orchestra.run.smithi193.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-09-16T00:58:44.144 INFO:teuthology.orchestra.run.smithi193.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-16T00:58:44.160 INFO:teuthology.orchestra.run.smithi148.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 673 kB/s | 45 kB 00:00 2024-09-16T00:58:44.170 INFO:teuthology.orchestra.run.smithi193.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 716 kB/s | 42 kB 00:00 2024-09-16T00:58:44.185 INFO:teuthology.orchestra.run.smithi148.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 476 kB/s | 64 kB 00:00 2024-09-16T00:58:44.195 INFO:teuthology.orchestra.run.smithi193.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-09-16T00:58:44.220 INFO:teuthology.orchestra.run.smithi193.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-16T00:58:44.246 INFO:teuthology.orchestra.run.smithi193.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-16T00:58:44.252 INFO:teuthology.orchestra.run.smithi148.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-09-16T00:58:44.271 INFO:teuthology.orchestra.run.smithi193.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T00:58:44.296 INFO:teuthology.orchestra.run.smithi193.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-16T00:58:44.311 INFO:teuthology.orchestra.run.smithi148.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-16T00:58:44.336 INFO:teuthology.orchestra.run.smithi148.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 12 MB/s | 308 kB 00:00 2024-09-16T00:58:44.346 INFO:teuthology.orchestra.run.smithi193.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 834 kB/s | 42 kB 00:00 2024-09-16T00:58:44.361 INFO:teuthology.orchestra.run.smithi148.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-16T00:58:44.389 INFO:teuthology.orchestra.run.smithi193.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-16T00:58:44.395 INFO:teuthology.orchestra.run.smithi148.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-09-16T00:58:44.446 INFO:teuthology.orchestra.run.smithi148.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-09-16T00:58:44.447 INFO:teuthology.orchestra.run.smithi193.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-16T00:58:44.470 INFO:teuthology.orchestra.run.smithi148.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T00:58:44.481 INFO:teuthology.orchestra.run.smithi193.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 1.1 MB/s | 36 kB 00:00 2024-09-16T00:58:44.496 INFO:teuthology.orchestra.run.smithi148.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-16T00:58:44.530 INFO:teuthology.orchestra.run.smithi148.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-16T00:58:44.613 INFO:teuthology.orchestra.run.smithi148.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-09-16T00:58:44.647 INFO:teuthology.orchestra.run.smithi148.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T00:58:44.648 INFO:teuthology.orchestra.run.smithi193.stdout:(96/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-16T00:58:44.664 INFO:teuthology.orchestra.run.smithi148.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.5 MB/s | 43 kB 00:00 2024-09-16T00:58:44.664 INFO:teuthology.orchestra.run.smithi193.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-09-16T00:58:44.690 INFO:teuthology.orchestra.run.smithi193.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 7.1 MB/s | 304 kB 00:00 2024-09-16T00:58:44.697 INFO:teuthology.orchestra.run.smithi148.stdout:(115/156): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-09-16T00:58:44.707 INFO:teuthology.orchestra.run.smithi193.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2024-09-16T00:58:44.714 INFO:teuthology.orchestra.run.smithi148.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-09-16T00:58:44.734 INFO:teuthology.orchestra.run.smithi193.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 2.2 MB/s | 96 kB 00:00 2024-09-16T00:58:44.734 INFO:teuthology.orchestra.run.smithi148.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 422 kB/s | 14 kB 00:00 2024-09-16T00:58:44.749 INFO:teuthology.orchestra.run.smithi193.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T00:58:44.756 INFO:teuthology.orchestra.run.smithi148.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-16T00:58:44.766 INFO:teuthology.orchestra.run.smithi193.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-09-16T00:58:44.782 INFO:teuthology.orchestra.run.smithi148.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-16T00:58:44.815 INFO:teuthology.orchestra.run.smithi148.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-16T00:58:44.841 INFO:teuthology.orchestra.run.smithi148.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-16T00:58:44.866 INFO:teuthology.orchestra.run.smithi193.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-16T00:58:44.874 INFO:teuthology.orchestra.run.smithi148.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-16T00:58:44.899 INFO:teuthology.orchestra.run.smithi148.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-16T00:58:44.917 INFO:teuthology.orchestra.run.smithi193.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-16T00:58:44.925 INFO:teuthology.orchestra.run.smithi148.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 709 kB/s | 18 kB 00:00 2024-09-16T00:58:44.950 INFO:teuthology.orchestra.run.smithi148.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 930 kB/s | 23 kB 00:00 2024-09-16T00:58:44.951 INFO:teuthology.orchestra.run.smithi193.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.0 MB/s | 308 kB 00:00 2024-09-16T00:58:44.976 INFO:teuthology.orchestra.run.smithi193.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-16T00:58:44.984 INFO:teuthology.orchestra.run.smithi148.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 580 kB/s | 19 kB 00:00 2024-09-16T00:58:45.009 INFO:teuthology.orchestra.run.smithi148.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 798 kB/s | 20 kB 00:00 2024-09-16T00:58:45.051 INFO:teuthology.orchestra.run.smithi193.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-09-16T00:58:45.059 INFO:teuthology.orchestra.run.smithi148.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-09-16T00:58:45.085 INFO:teuthology.orchestra.run.smithi148.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T00:58:45.110 INFO:teuthology.orchestra.run.smithi148.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T00:58:45.118 INFO:teuthology.orchestra.run.smithi193.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-09-16T00:58:45.136 INFO:teuthology.orchestra.run.smithi148.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-16T00:58:45.143 INFO:teuthology.orchestra.run.smithi193.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-16T00:58:45.168 INFO:teuthology.orchestra.run.smithi193.stdout:(110/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-09-16T00:58:45.169 INFO:teuthology.orchestra.run.smithi148.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-16T00:58:45.194 INFO:teuthology.orchestra.run.smithi193.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T00:58:45.194 INFO:teuthology.orchestra.run.smithi148.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 662 kB/s | 16 kB 00:00 2024-09-16T00:58:45.219 INFO:teuthology.orchestra.run.smithi193.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-16T00:58:45.228 INFO:teuthology.orchestra.run.smithi148.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-09-16T00:58:45.253 INFO:teuthology.orchestra.run.smithi193.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-16T00:58:45.253 INFO:teuthology.orchestra.run.smithi148.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-16T00:58:45.278 INFO:teuthology.orchestra.run.smithi193.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T00:58:45.300 INFO:teuthology.orchestra.run.smithi148.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 394 kB/s | 28 kB 00:00 2024-09-16T00:58:45.304 INFO:teuthology.orchestra.run.smithi193.stdout:(115/156): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-09-16T00:58:45.329 INFO:teuthology.orchestra.run.smithi193.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-09-16T00:58:45.329 INFO:teuthology.orchestra.run.smithi148.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 1.1 MB/s | 31 kB 00:00 2024-09-16T00:58:45.354 INFO:teuthology.orchestra.run.smithi148.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.5 MB/s | 188 kB 00:00 2024-09-16T00:58:45.354 INFO:teuthology.orchestra.run.smithi193.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-16T00:58:45.379 INFO:teuthology.orchestra.run.smithi148.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-16T00:58:45.400 INFO:teuthology.orchestra.run.smithi193.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.4 MB/s | 172 kB 00:00 2024-09-16T00:58:45.404 INFO:teuthology.orchestra.run.smithi148.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-09-16T00:58:45.430 INFO:teuthology.orchestra.run.smithi148.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-16T00:58:45.430 INFO:teuthology.orchestra.run.smithi193.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-09-16T00:58:45.455 INFO:teuthology.orchestra.run.smithi148.stdout:(142/156): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T00:58:45.455 INFO:teuthology.orchestra.run.smithi193.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.4 MB/s | 303 kB 00:00 2024-09-16T00:58:45.481 INFO:teuthology.orchestra.run.smithi148.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T00:58:45.481 INFO:teuthology.orchestra.run.smithi193.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-16T00:58:45.506 INFO:teuthology.orchestra.run.smithi148.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 801 kB/s | 20 kB 00:00 2024-09-16T00:58:45.514 INFO:teuthology.orchestra.run.smithi193.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-16T00:58:45.531 INFO:teuthology.orchestra.run.smithi148.stdout:(145/156): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-16T00:58:45.539 INFO:teuthology.orchestra.run.smithi193.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-09-16T00:58:45.565 INFO:teuthology.orchestra.run.smithi148.stdout:(146/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-09-16T00:58:45.565 INFO:teuthology.orchestra.run.smithi193.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 709 kB/s | 18 kB 00:00 2024-09-16T00:58:45.591 INFO:teuthology.orchestra.run.smithi193.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-09-16T00:58:45.615 INFO:teuthology.orchestra.run.smithi193.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 779 kB/s | 19 kB 00:00 2024-09-16T00:58:45.623 INFO:teuthology.orchestra.run.smithi148.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-16T00:58:45.641 INFO:teuthology.orchestra.run.smithi193.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 786 kB/s | 20 kB 00:00 2024-09-16T00:58:45.691 INFO:teuthology.orchestra.run.smithi193.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-09-16T00:58:45.798 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-09-16T00:58:45.798 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-09-16T00:58:45.824 INFO:teuthology.orchestra.run.smithi148.stdout:(148/156): librados2-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 3.4 MB 00:00 2024-09-16T00:58:45.974 INFO:teuthology.orchestra.run.smithi148.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 9.0 MB/s | 3.2 MB 00:00 2024-09-16T00:58:46.142 INFO:teuthology.orchestra.run.smithi193.stdout:(129/156): ceph-test-19.1.1-326.g8921c8ef.el9.x 5.0 MB/s | 49 MB 00:09 2024-09-16T00:58:46.149 INFO:teuthology.orchestra.run.smithi148.stdout:(150/156): NetworkManager-1.49.90-1.el9.x86_64. 7.1 MB/s | 2.3 MB 00:00 2024-09-16T00:58:46.167 INFO:teuthology.orchestra.run.smithi193.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 97 kB/s | 46 kB 00:00 2024-09-16T00:58:46.191 INFO:teuthology.orchestra.run.smithi148.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 981 kB/s | 41 kB 00:00 2024-09-16T00:58:46.200 INFO:teuthology.orchestra.run.smithi193.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-09-16T00:58:46.234 INFO:teuthology.orchestra.run.smithi193.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 8.5 MB/s | 272 kB 00:00 2024-09-16T00:58:46.250 INFO:teuthology.orchestra.run.smithi148.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 4.2 MB/s | 249 kB 00:00 2024-09-16T00:58:46.259 INFO:teuthology.orchestra.run.smithi193.stdout:(133/156): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-09-16T00:58:46.276 INFO:teuthology.orchestra.run.smithi193.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 391 kB/s | 16 kB 00:00 2024-09-16T00:58:46.293 INFO:teuthology.orchestra.run.smithi193.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-09-16T00:58:46.318 INFO:teuthology.orchestra.run.smithi193.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-16T00:58:46.335 INFO:teuthology.orchestra.run.smithi193.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 735 kB/s | 31 kB 00:00 2024-09-16T00:58:46.369 INFO:teuthology.orchestra.run.smithi193.stdout:(138/156): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-09-16T00:58:46.403 INFO:teuthology.orchestra.run.smithi193.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 878 kB/s | 59 kB 00:00 2024-09-16T00:58:46.408 INFO:teuthology.orchestra.run.smithi148.stdout:(153/156): NetworkManager-libnm-1.49.90-1.el9.x 4.2 MB/s | 1.8 MB 00:00 2024-09-16T00:58:46.428 INFO:teuthology.orchestra.run.smithi193.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 608 kB/s | 36 kB 00:00 2024-09-16T00:58:46.453 INFO:teuthology.orchestra.run.smithi193.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-09-16T00:58:46.467 INFO:teuthology.orchestra.run.smithi148.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 6.2 MB/s | 1.4 MB 00:00 2024-09-16T00:58:46.479 INFO:teuthology.orchestra.run.smithi193.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 989 kB/s | 74 kB 00:00 2024-09-16T00:58:46.504 INFO:teuthology.orchestra.run.smithi193.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T00:58:46.529 INFO:teuthology.orchestra.run.smithi193.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 803 kB/s | 20 kB 00:00 2024-09-16T00:58:46.555 INFO:teuthology.orchestra.run.smithi193.stdout:(145/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-16T00:58:46.588 INFO:teuthology.orchestra.run.smithi193.stdout:(146/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-09-16T00:58:46.613 INFO:teuthology.orchestra.run.smithi193.stdout:(147/156): re2-20211101-20.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-16T00:58:46.664 INFO:teuthology.orchestra.run.smithi193.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T00:58:46.901 INFO:teuthology.orchestra.run.smithi148.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 4.9 MB/s | 2.4 MB 00:00 2024-09-16T00:58:46.989 INFO:teuthology.orchestra.run.smithi193.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 7.1 MB/s | 2.3 MB 00:00 2024-09-16T00:58:47.190 INFO:teuthology.orchestra.run.smithi193.stdout:(150/156): NetworkManager-libnm-1.49.90-1.el9.x 9.0 MB/s | 1.8 MB 00:00 2024-09-16T00:58:47.248 INFO:teuthology.orchestra.run.smithi193.stdout:(151/156): librados2-19.1.1-326.g8921c8ef.el9.x 5.2 MB/s | 3.4 MB 00:00 2024-09-16T00:58:47.307 INFO:teuthology.orchestra.run.smithi193.stdout:(152/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 4.6 MB/s | 3.2 MB 00:00 2024-09-16T00:58:47.348 INFO:teuthology.orchestra.run.smithi193.stdout:(153/156): NetworkManager-team-1.49.90-1.el9.x8 257 kB/s | 41 kB 00:00 2024-09-16T00:58:47.374 INFO:teuthology.orchestra.run.smithi193.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 1.9 MB/s | 249 kB 00:00 2024-09-16T00:58:47.691 INFO:teuthology.orchestra.run.smithi193.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 3.5 MB/s | 1.4 MB 00:00 2024-09-16T00:58:47.733 INFO:teuthology.orchestra.run.smithi193.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 6.2 MB/s | 2.4 MB 00:00 2024-09-16T00:58:47.740 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:58:47.740 INFO:teuthology.orchestra.run.smithi193.stdout:Total 13 MB/s | 226 MB 00:17 2024-09-16T00:58:48.984 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-09-16T00:58:49.134 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-09-16T00:58:49.135 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-09-16T00:58:49.446 INFO:teuthology.orchestra.run.smithi148.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:06 2024-09-16T00:58:49.451 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T00:58:49.451 INFO:teuthology.orchestra.run.smithi148.stdout:Total 12 MB/s | 226 MB 00:18 2024-09-16T00:58:50.700 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T00:58:50.855 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T00:58:50.855 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T00:58:50.938 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-09-16T00:58:50.939 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-09-16T00:58:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-09-16T00:58:52.382 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:58:52.430 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:58:52.482 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:58:52.515 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:58:52.549 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:58:52.605 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:58:52.700 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T00:58:52.701 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T00:58:52.955 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:58:53.067 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:58:53.191 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:58:53.218 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:58:53.293 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:58:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:58:53.519 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:58:53.585 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:58:53.614 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:58:53.720 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:58:53.740 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:58:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:58:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:58:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:58:54.068 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:58:54.233 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:58:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:58:54.454 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:54.782 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:58:54.992 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:58:55.030 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:58:55.081 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:58:55.145 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:58:55.190 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:58:55.222 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:58:55.266 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:58:55.299 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:58:55.326 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:58:55.394 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:58:55.465 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:58:55.496 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:58:55.539 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:58:55.614 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:58:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:58:55.694 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:58:55.722 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:58:55.791 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:58:55.845 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:58:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:58:56.060 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:58:56.113 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:58:56.632 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:58:56.709 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:58:56.738 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:58:56.781 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:58:56.819 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:58:56.861 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:58:57.022 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:58:57.072 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:58:57.127 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:58:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:58:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:58:57.267 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:58:57.314 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:58:57.359 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:58:57.410 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:58:57.452 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:58:57.471 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-09-16T00:58:57.553 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:58:57.586 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:58:57.591 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:58:57.629 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:58:57.645 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:58:57.670 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:58:57.679 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:58:57.704 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:58:57.738 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:58:57.775 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:58:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:58:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:58:57.914 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:58:57.939 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:58:58.063 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:58:58.069 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:58:58.180 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:58:58.303 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:58:58.331 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:58:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:58:58.452 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:58:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:58:58.605 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:58:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:58:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:58:58.790 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:58:58.820 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:58:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:58:58.883 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:58:58.905 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:58:58.917 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:58:58.932 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:58:58.960 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:58:58.973 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T00:58:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:58:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:58:59.073 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:58:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:58:59.087 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T00:58:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:58:59.126 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T00:58:59.144 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:58:59.188 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T00:58:59.195 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:58:59.228 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:58:59.229 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:58:59.230 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T00:58:59.318 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:58:59.344 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T00:58:59.353 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:58:59.368 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:58:59.398 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:58:59.455 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:58:59.530 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:58:59.577 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T00:58:59.768 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:58:59.890 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:58:59.942 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T00:58:59.982 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:59:00.010 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:59:00.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:59:00.048 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:59:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:59:00.163 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:59:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:59:00.191 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:59:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:59:00.247 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:59:00.279 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:59:00.282 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:59:00.288 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:59:00.307 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:59:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:59:00.338 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:59:00.372 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:59:00.375 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:59:00.429 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:59:00.459 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:59:00.466 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T00:59:00.471 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:59:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:59:00.525 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T00:59:00.528 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:59:00.553 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:59:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:59:00.580 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:59:00.601 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:59:00.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:59:00.658 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:59:00.673 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:59:00.676 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:59:00.687 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:59:00.725 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:59:00.742 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T00:59:00.743 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:59:00.764 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:59:00.787 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T00:59:00.797 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:59:00.871 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:59:00.908 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:59:00.940 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:59:00.940 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:59:01.011 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:59:01.017 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:59:01.065 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:59:01.066 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T00:59:01.178 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T00:59:01.268 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:59:01.371 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:59:01.584 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:59:01.597 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T00:59:01.656 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:59:01.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:59:01.690 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:59:01.713 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T00:59:01.733 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:59:01.770 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:59:01.771 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:59:01.809 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T00:59:01.812 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:59:01.848 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T00:59:01.881 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T00:59:01.956 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T00:59:01.970 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:59:02.008 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T00:59:02.024 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:59:02.048 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T00:59:02.079 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:59:02.100 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T00:59:02.140 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T00:59:02.160 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:59:02.168 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T00:59:02.192 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:59:02.228 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:59:02.238 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T00:59:02.275 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:59:02.300 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T00:59:02.312 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:59:02.329 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T00:59:02.354 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:59:02.373 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T00:59:02.397 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:59:02.449 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T00:59:02.483 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T00:59:02.527 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T00:59:02.532 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:59:02.564 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T00:59:02.586 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:59:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:59:02.636 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T00:59:02.687 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T00:59:02.730 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:59:02.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:59:02.771 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T00:59:02.871 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:59:02.907 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T00:59:02.957 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T00:59:03.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:59:03.018 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:59:03.034 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:59:03.094 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:59:03.104 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:59:03.122 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:59:03.154 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:59:03.245 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:59:03.284 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:59:03.312 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:59:03.346 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:59:03.385 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:59:03.417 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:59:03.447 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:59:03.475 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:59:03.481 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:59:03.503 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T00:59:03.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:59:03.517 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:03.556 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:59:03.586 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T00:59:03.599 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:59:03.607 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:03.614 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T00:59:03.639 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:59:03.656 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T00:59:03.675 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:59:03.685 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:59:03.694 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T00:59:03.730 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:59:03.737 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T00:59:03.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:59:03.864 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:59:03.913 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T00:59:03.920 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:59:03.973 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T00:59:04.028 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T00:59:04.102 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T00:59:04.132 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T00:59:04.157 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:59:04.169 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T00:59:04.188 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:04.224 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T00:59:04.259 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:04.268 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T00:59:04.298 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:59:04.311 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T00:59:04.354 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T00:59:04.491 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T00:59:04.538 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T00:59:04.579 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T00:59:04.639 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:59:04.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T00:59:04.684 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:59:04.718 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:59:04.721 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T00:59:04.761 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T00:59:04.786 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T00:59:04.820 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T00:59:04.821 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:59:04.884 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:59:04.896 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T00:59:04.929 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:59:05.016 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:59:05.022 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T00:59:05.048 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:59:05.060 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T00:59:05.088 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T00:59:05.122 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T00:59:05.161 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T00:59:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T00:59:05.233 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:59:05.239 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T00:59:05.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:59:05.274 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T00:59:05.307 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T00:59:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:59:05.349 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T00:59:05.383 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T00:59:05.471 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T00:59:05.522 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T00:59:05.553 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T00:59:05.666 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T00:59:05.721 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T00:59:05.975 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:59:06.078 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:59:06.462 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T00:59:06.518 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T00:59:06.551 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T00:59:06.642 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:06.654 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T00:59:06.687 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:06.715 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T00:59:06.760 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T00:59:06.849 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T00:59:06.888 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T00:59:07.071 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T00:59:07.101 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T00:59:07.179 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T00:59:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:07.290 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:07.300 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:59:07.375 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:07.375 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:59:07.375 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:59:07.375 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:07.385 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:59:07.409 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:07.412 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:59:07.446 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:59:07.765 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:59:07.809 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:07.840 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T00:59:07.908 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:07.939 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:59:07.951 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T00:59:07.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:59:08.457 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:59:08.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:08.551 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:08.580 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:59:09.202 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T00:59:09.282 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T00:59:09.319 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T00:59:09.360 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T00:59:09.680 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T00:59:09.731 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:09.822 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T00:59:09.852 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T00:59:09.888 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T00:59:10.374 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T00:59:10.413 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:10.488 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T00:59:10.518 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T00:59:10.867 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:10.902 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:11.447 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:11.489 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:11.588 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:11.588 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:59:11.588 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:59:11.589 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:11.624 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:12.841 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:12.873 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:13.425 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T00:59:13.460 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:13.567 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T00:59:13.567 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T00:59:13.567 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T00:59:13.567 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:13.603 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-09-16T00:59:19.160 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-09-16T00:59:19.161 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-09-16T00:59:19.161 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-09-16T00:59:19.161 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:19.337 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:19.353 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:19.354 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:19.354 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:59:19.354 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:20.823 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:20.851 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:21.082 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:21.105 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:21.106 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:21.106 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:59:21.106 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:21.106 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:21.106 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:21.550 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:21.573 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:21.573 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:21.574 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:59:21.574 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:21.574 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:21.574 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:21.680 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:59:21.736 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:59:21.791 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:59:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:59:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:59:22.111 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:59:22.208 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:59:22.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:59:22.399 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:59:22.433 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:59:22.460 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:59:22.923 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:22.951 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:22.982 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:59:23.045 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:59:23.074 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-09-16T00:59:23.298 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-09-16T00:59:23.299 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-09-16T00:59:23.299 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:23.471 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:23.487 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:23.488 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:23.488 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:59:23.488 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:24.118 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:24.151 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:24.933 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:24.960 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:25.119 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:25.151 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:25.183 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:25.203 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /root 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2024-09-16T00:59:25.204 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:25.207 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:25.207 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:25.208 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:59:25.208 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:25.208 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:25.208 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:25.254 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:25.381 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:25.397 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T00:59:25.397 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:25.397 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T00:59:25.398 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:59:25.454 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:25.478 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:25.529 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:25.541 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:25.637 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:59:25.643 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:25.666 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:25.666 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:25.666 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:59:25.666 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:25.666 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:25.667 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:25.696 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:59:25.773 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:59:25.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:59:25.885 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:59:26.072 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:59:26.128 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:59:26.205 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:59:26.294 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:59:26.435 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:59:26.486 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:59:26.518 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:59:26.545 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:59:26.622 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:59:26.656 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:26.680 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:26.680 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:26.681 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:59:26.681 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:26.681 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:26.681 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:26.747 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:26.747 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:59:26.747 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:26.791 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:26.817 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:26.892 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T00:59:26.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:26.975 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:27.003 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:27.034 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:59:27.097 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:59:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:27.126 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:59:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:27.151 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T00:59:27.175 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:27.618 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T00:59:27.643 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:27.749 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T00:59:27.814 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T00:59:27.878 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T00:59:28.066 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T00:59:28.119 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T00:59:28.177 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:28.198 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T00:59:28.211 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:28.311 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T00:59:28.453 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T00:59:28.511 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T00:59:28.543 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T00:59:28.570 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T00:59:28.916 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:28.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T00:59:28.967 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T00:59:29.029 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T00:59:29.058 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:29.177 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:29.203 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:29.307 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:29.475 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:59:29.514 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:29.539 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:29.539 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:29.539 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:59:29.539 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:29.540 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:29.540 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:29.590 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:29.602 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:29.698 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:59:29.746 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:59:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T00:59:30.160 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:30.677 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:59:30.709 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:30.732 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:30.733 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:30.733 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:59:30.733 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:30.733 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:30.733 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:30.776 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:30.799 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:30.799 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:30.799 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:59:30.800 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:30.852 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:30.876 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:31.146 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T00:59:31.169 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:31.177 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:31.201 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:31.201 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:31.202 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:59:31.202 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:31.202 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:31.202 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:31.275 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T00:59:31.444 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T00:59:31.471 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:31.496 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T00:59:31.496 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:31.497 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T00:59:31.497 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:31.497 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T00:59:31.497 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:31.547 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:31.561 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T00:59:31.650 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T00:59:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:59:31.704 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T00:59:31.728 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:59:31.777 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:59:31.809 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:59:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:59:32.030 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:59:32.076 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:59:32.122 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:59:32.160 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:59:32.206 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:59:32.653 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T00:59:32.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T00:59:32.707 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:32.750 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:32.773 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T00:59:32.774 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:32.774 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T00:59:32.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:32.804 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:32.834 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:32.838 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:32.858 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T00:59:32.858 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:32.858 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T00:59:32.859 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:32.859 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T00:59:32.859 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:32.859 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:33.168 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:33.192 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T00:59:33.192 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T00:59:33.192 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T00:59:33.192 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:33.193 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T00:59:33.193 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:33.678 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:59:33.768 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:59:33.817 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:59:33.906 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:59:33.937 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:59:33.963 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:59:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:59:33.997 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:34.089 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:34.090 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:34.157 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:34.182 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:59:34.187 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:34.187 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:34.367 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:34.367 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:34.451 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:35.748 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:59:35.789 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:59:35.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:59:35.863 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:59:35.890 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:59:36.079 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:59:36.129 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:59:36.159 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:59:36.189 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:59:36.235 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:59:36.848 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:36.883 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:36.905 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:37.743 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:59:37.821 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T00:59:37.839 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:59:37.863 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T00:59:37.879 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:59:37.904 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T00:59:37.935 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T00:59:37.962 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:59:37.988 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T00:59:37.992 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:59:38.018 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:59:38.043 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:59:38.044 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:38.178 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T00:59:38.220 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:38.226 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T00:59:38.245 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:59:38.251 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:38.251 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:38.256 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T00:59:38.286 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T00:59:38.324 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T00:59:38.430 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:38.430 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:38.498 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:38.572 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:38.949 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:38.997 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:39.018 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T00:59:39.872 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T00:59:39.968 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T00:59:40.009 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T00:59:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:59:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:59:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:59:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:59:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:59:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:59:40.096 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T00:59:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:59:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:59:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:59:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:59:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:59:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:59:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:59:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:59:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:59:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:59:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:59:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:59:40.121 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T00:59:40.146 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T00:59:40.172 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T00:59:40.172 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:40.264 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T00:59:40.265 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:40.348 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T00:59:40.373 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T00:59:40.379 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:40.379 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:40.559 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T00:59:40.559 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:40.643 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T00:59:40.706 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:42.655 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:42.655 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:59:42.656 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.657 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.658 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:59:42.659 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.660 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:59:42.661 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:59:42.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:59:42.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:59:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:59:42.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:59:42.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:59:42.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:59:42.668 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:59:42.669 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:59:42.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T00:59:42.669 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-09-16T00:59:43.104 DEBUG:teuthology.parallel:result is None 2024-09-16T00:59:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:59:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:59:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:59:44.276 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:59:44.277 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:59:44.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:59:44.279 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:59:44.280 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:59:44.281 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:59:44.282 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:59:44.283 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:59:44.284 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:59:44.285 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:59:44.286 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:59:44.287 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:59:44.288 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:59:44.289 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:59:44.289 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:59:44.289 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:59:46.450 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T00:59:46.451 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T00:59:46.452 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T00:59:46.453 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T00:59:46.454 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T00:59:46.455 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T00:59:46.456 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T00:59:46.457 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T00:59:46.458 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T00:59:46.459 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T00:59:46.460 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T00:59:46.461 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T00:59:46.462 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T00:59:46.463 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T00:59:46.464 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:46.868 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.869 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:59:46.871 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:59:46.872 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:59:46.873 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:59:46.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:59:46.875 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:59:46.876 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:59:46.877 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:59:46.878 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:59:46.879 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:59:46.880 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T00:59:46.881 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-09-16T00:59:47.289 DEBUG:teuthology.parallel:result is None 2024-09-16T00:59:48.915 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:48.916 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T00:59:48.917 INFO:teuthology.orchestra.run.smithi148.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.918 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.919 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.920 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T00:59:48.921 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T00:59:48.922 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T00:59:48.923 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T00:59:48.924 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T00:59:48.925 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T00:59:48.926 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T00:59:48.927 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T00:59:48.927 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T00:59:48.927 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T00:59:48.928 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T00:59:48.929 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T00:59:48.930 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:48.931 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T00:59:48.932 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T00:59:48.933 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.934 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T00:59:48.935 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T00:59:48.936 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T00:59:48.937 INFO:teuthology.orchestra.run.smithi148.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T00:59:48.938 INFO:teuthology.orchestra.run.smithi148.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T00:59:48.938 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T00:59:48.938 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T00:59:48.938 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T00:59:48.938 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T00:59:48.939 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T00:59:49.317 DEBUG:teuthology.parallel:result is None 2024-09-16T00:59:49.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:49.465 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:59:49.492 INFO:teuthology.orchestra.run.smithi078.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.493 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.493 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:59:49.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:49.635 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:59:49.657 INFO:teuthology.orchestra.run.smithi148.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.658 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.658 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:59:49.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:49.791 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T00:59:49.817 INFO:teuthology.orchestra.run.smithi193.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.817 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T00:59:49.818 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T00:59:49.819 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T00:59:49.820 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:59:49.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:59:49.853 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:59:49.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:59:49.880 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:59:49.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T00:59:49.911 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T00:59:49.912 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:59:49.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:59:49.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:59:50.010 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:59:50.010 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:59:50.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:59:50.100 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:59:50.100 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T00:59:50.129 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T00:59:50.198 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T00:59:50.199 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:59:50.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:59:50.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:59:50.296 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:59:50.296 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:59:50.320 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:59:50.385 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:59:50.386 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T00:59:50.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T00:59:50.485 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T00:59:50.761 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T00:59:50.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:59:50.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:59:50.861 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T00:59:50.861 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:59:50.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:59:50.952 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T00:59:50.952 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T00:59:50.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T00:59:51.051 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-16T00:59:51.145 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-16T00:59:51.145 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:51.146 INFO:tasks.cephadm:Cluster fsid is fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T00:59:51.146 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-16T00:59:51.146 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.148', 'mon.c': '172.21.15.193'} 2024-09-16T00:59:51.146 INFO:tasks.cephadm:First mon is mon.a on smithi078 2024-09-16T00:59:51.147 INFO:tasks.cephadm:First mgr is x 2024-09-16T00:59:51.147 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-16T00:59:51.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-09-16T00:59:51.177 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname $(hostname -s) 2024-09-16T00:59:51.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-09-16T00:59:51.232 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-16T00:59:51.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:51.389 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-16T00:59:51.529 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T00:59:51.530 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T00:59:51.530 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-16T00:59:51.530 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:51.875 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:59 /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:51.875 DEBUG:teuthology.orchestra.run.smithi148:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.256 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:59 /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.257 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.648 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 16 00:59 /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.648 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.668 DEBUG:teuthology.orchestra.run.smithi148:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.688 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-16T00:59:52.724 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-16T00:59:52.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:59:52.727 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:59:52.730 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-16T00:59:54.205 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:59:54.335 INFO:teuthology.orchestra.run.smithi148.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T00:59:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T01:00:34.409 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout: "repo_digests": [ 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T01:00:34.410 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T01:00:35.090 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-09-16T01:00:35.090 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T01:00:35.090 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T01:00:35.090 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-09-16T01:00:35.090 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T01:00:35.091 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-09-16T01:00:35.091 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-09-16T01:00:36.124 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:00:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T01:00:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-16T01:00:36.124 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2024-09-16T01:00:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-16T01:00:36.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:00:36.125 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:00:36.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-09-16T01:00:36.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph 2024-09-16T01:00:36.235 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-09-16T01:00:36.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-09-16T01:00:36.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /etc/ceph 2024-09-16T01:00:36.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-09-16T01:00:36.351 INFO:tasks.cephadm:Writing seed config... 2024-09-16T01:00:36.352 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-16T01:00:36.352 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-16T01:00:36.352 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-16T01:00:36.352 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-16T01:00:36.353 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-16T01:00:36.354 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-16T01:00:36.355 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-16T01:00:36.356 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-16T01:00:36.357 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:00:36.357 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-16T01:00:36.372 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = fa4731de-73c6-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-16T01:00:36.372 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service 2024-09-16T01:00:36.414 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x.service 2024-09-16T01:00:36.457 INFO:tasks.cephadm:Bootstrapping... 2024-09-16T01:00:36.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:00:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:00:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', 'fa4731de-73c6-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2024-09-16T01:00:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-16T01:00:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2024-09-16T01:00:36.684 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 5.2.2 2024-09-16T01:00:36.684 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2024-09-16T01:00:36.684 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2024-09-16T01:00:36.691 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T01:00:36.691 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T01:00:36.697 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T01:00:36.697 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:00:36.704 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-09-16T01:00:36.710 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-09-16T01:00:36.710 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-09-16T01:00:36.711 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2024-09-16T01:00:36.732 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 5.2.2 2024-09-16T01:00:36.733 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-16T01:00:36.733 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2024-09-16T01:00:36.733 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2024-09-16T01:00:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-16T01:00:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-16T01:00:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-16T01:00:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:00:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140422778143744 on /run/cephadm/fa4731de-73c6-11ef-bceb-c7b262605968.lock 2024-09-16T01:00:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140422778143744 acquired on /run/cephadm/fa4731de-73c6-11ef-bceb-c7b262605968.lock 2024-09-16T01:00:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2024-09-16T01:00:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2024-09-16T01:00:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-09-16T01:00:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.78 metric 100 2024-09-16T01:00:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2024-09-16T01:00:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-16T01:00:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-16T01:00:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-16T01:00:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-16T01:00:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-16T01:00:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T01:00:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-16T01:00:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link noprefixroute 2024-09-16T01:00:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-16T01:00:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-09-16T01:00:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-09-16T01:00:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-16T01:00:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-16T01:00:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-16T01:00:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-16T01:00:38.348 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:00:38.349 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:00:38.349 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2024-09-16T01:00:39.354 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2024-09-16T01:00:39.354 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2024-09-16T01:00:40.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQA3g+dmBp/gJxAAxe1dGLLvlMv1HxGqjYpCIg== 2024-09-16T01:00:41.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQA4g+dm22E2KRAA+UUOI/PCWgzeSkvQBaZDEw== 2024-09-16T01:00:42.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQA5g+dmJ3adIxAACtVSONcccffXZ99GnucJzw== 2024-09-16T01:00:42.212 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:00:43.068 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2024-09-16T01:00:43.914 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2024-09-16T01:00:44.129 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-16T01:00:44.273 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-16T01:00:44.452 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target. 2024-09-16T01:00:44.453 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target. 2024-09-16T01:00:44.718 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a 2024-09-16T01:00:44.719 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-16T01:00:44.884 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target.wants/ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968@.service. 2024-09-16T01:00:45.388 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T01:00:45.388 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-16T01:00:45.394 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T01:00:45.394 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:00:45.394 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-16T01:00:45.394 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T01:00:45.395 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2024-09-16T01:00:45.395 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2024-09-16T01:00:46.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:46 smithi078 ceph-mon[30575]: from='client.? 172.21.15.78:0/671720800' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-16T01:00:46.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2024-09-16T01:00:46.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.532673s) 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-16T01:00:46.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7f1256640 1 Processor -- start 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7f1256640 1 -- start start 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7f1256640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7f1256640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec107070 con 0x7ff7ec1066a0 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7ead9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7ead9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37572/0 (socket says 172.21.15.78:37572) 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7ead9d640 1 -- 172.21.15.78:0/3024959051 learned_addr learned my addr 172.21.15.78:0/3024959051 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.811+0000 7ff7ead9d640 1 -- 172.21.15.78:0/3024959051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec107850 con 0x7ff7ec1066a0 2024-09-16T01:00:46.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.813+0000 7ff7ead9d640 1 --2- 172.21.15.78:0/3024959051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009820 tx=0x7ff7dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f64488f441fc004 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.813+0000 7ff7e9d9b640 1 -- 172.21.15.78:0/3024959051 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7dc037070 con 0x7ff7ec1066a0 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.813+0000 7ff7e9d9b640 1 -- 172.21.15.78:0/3024959051 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc032440 con 0x7ff7ec1066a0 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.813+0000 7ff7e9d9b640 1 -- 172.21.15.78:0/3024959051 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7dc032740 con 0x7ff7ec1066a0 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 -- 172.21.15.78:0/3024959051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 msgr2=0x7ff7ec106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 --2- 172.21.15.78:0/3024959051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009820 tx=0x7ff7dc02a270 comp rx=0 tx=0).stop 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 -- 172.21.15.78:0/3024959051 shutdown_connections 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 --2- 172.21.15.78:0/3024959051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec106aa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 -- 172.21.15.78:0/3024959051 >> 172.21.15.78:0/3024959051 conn(0x7ff7ec101ed0 msgr2=0x7ff7ec1042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 -- 172.21.15.78:0/3024959051 shutdown_connections 2024-09-16T01:00:46.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.814+0000 7ff7f1256640 1 -- 172.21.15.78:0/3024959051 wait complete. 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7f1256640 1 Processor -- start 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7f1256640 1 -- start start 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7f1256640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7f1256640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec107df0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7ead9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7ead9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37574/0 (socket says 172.21.15.78:37574) 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.815+0000 7ff7ead9d640 1 -- 172.21.15.78:0/671720800 learned_addr learned my addr 172.21.15.78:0/671720800 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7ead9d640 1 -- 172.21.15.78:0/671720800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec19a330 con 0x7ff7ec1066a0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7ead9d640 1 --2- 172.21.15.78:0/671720800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0090b0 tx=0x7ff7dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7dc039600 con 0x7ff7ec1066a0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0397a0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec19a5c0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff7dc039ac0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec07b830 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff7dc04a060 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.816+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff7dc043730 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.817+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec108440 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.819+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff7dc0632d0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.877+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7ff7ec07bdf0 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.877+0000 7ff7e3fff640 1 -- 172.21.15.78:0/671720800 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7ff7dc051030 con 0x7ff7ec1066a0 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 msgr2=0x7ff7ec199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 --2- 172.21.15.78:0/671720800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0090b0 tx=0x7ff7dc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 shutdown_connections 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 --2- 172.21.15.78:0/671720800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec1066a0 0x7ff7ec199df0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:46.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 >> 172.21.15.78:0/671720800 conn(0x7ff7ec101ed0 msgr2=0x7ff7ec18e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:46.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 shutdown_connections 2024-09-16T01:00:46.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:45.878+0000 7ff7f1256640 1 -- 172.21.15.78:0/671720800 wait complete. 2024-09-16T01:00:46.462 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2024-09-16T01:00:46.462 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2024-09-16T01:00:47.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:47.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T01:00:47.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-16T01:00:47.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.851+0000 7f49aeff0640 1 Processor -- start 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.851+0000 7f49aeff0640 1 -- start start 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.851+0000 7f49aeff0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.851+0000 7f49aeff0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f49a807bdd0 con 0x7f49a8108210 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.852+0000 7f49acd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.852+0000 7f49acd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37588/0 (socket says 172.21.15.78:37588) 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.852+0000 7f49acd8c640 1 -- 172.21.15.78:0/2770559756 learned_addr learned my addr 172.21.15.78:0/2770559756 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:47.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.852+0000 7f49acd8c640 1 -- 172.21.15.78:0/2770559756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a810ab40 con 0x7f49a8108210 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49acd8c640 1 --2- 172.21.15.78:0/2770559756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4990009820 tx=0x7f499002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6244b1d38edc4eae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49a77fe640 1 -- 172.21.15.78:0/2770559756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990037070 con 0x7f49a8108210 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49a77fe640 1 -- 172.21.15.78:0/2770559756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4990032440 con 0x7f49a8108210 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49a77fe640 1 -- 172.21.15.78:0/2770559756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990032740 con 0x7f49a8108210 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 -- 172.21.15.78:0/2770559756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 msgr2=0x7f49a810a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 --2- 172.21.15.78:0/2770559756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4990009820 tx=0x7f499002a270 comp rx=0 tx=0).stop 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 -- 172.21.15.78:0/2770559756 shutdown_connections 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 --2- 172.21.15.78:0/2770559756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a810a600 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 -- 172.21.15.78:0/2770559756 >> 172.21.15.78:0/2770559756 conn(0x7f49a8101db0 msgr2=0x7f49a8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 -- 172.21.15.78:0/2770559756 shutdown_connections 2024-09-16T01:00:47.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.853+0000 7f49aeff0640 1 -- 172.21.15.78:0/2770559756 wait complete. 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49aeff0640 1 Processor -- start 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49aeff0640 1 -- start start 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49aeff0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49aeff0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f49a810afc0 con 0x7f49a8108210 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49acd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49acd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37598/0 (socket says 172.21.15.78:37598) 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49acd8c640 1 -- 172.21.15.78:0/921300588 learned_addr learned my addr 172.21.15.78:0/921300588 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.854+0000 7f49acd8c640 1 -- 172.21.15.78:0/921300588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a819b430 con 0x7f49a8108210 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49acd8c640 1 --2- 172.21.15.78:0/921300588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f49900090b0 tx=0x7f49900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990041070 con 0x7f49a8108210 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499003c7a0 con 0x7f49a8108210 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4990037040 con 0x7f49a8108210 2024-09-16T01:00:47.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a819b6c0 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a819e3b0 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4990044070 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.855+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f499003d3f0 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.856+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a8108670 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.858+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4990046450 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.915+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f49a802d940 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.918+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f4990046630 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.919+0000 7f49a5ffb640 1 -- 172.21.15.78:0/921300588 <== mon.0 v2:172.21.15.78:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4990056070 con 0x7f49a8108210 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 msgr2=0x7f49a819aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 --2- 172.21.15.78:0/921300588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f49900090b0 tx=0x7f49900046c0 comp rx=0 tx=0).stop 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 shutdown_connections 2024-09-16T01:00:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 --2- 172.21.15.78:0/921300588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49a8108210 0x7f49a819aef0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:47.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 >> 172.21.15.78:0/921300588 conn(0x7f49a8101db0 msgr2=0x7f49a8102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:47.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 shutdown_connections 2024-09-16T01:00:47.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:46.920+0000 7f49aeff0640 1 -- 172.21.15.78:0/921300588 wait complete. 2024-09-16T01:00:47.503 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2024-09-16T01:00:47.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:47 smithi078 ceph-mon[30575]: from='client.? 172.21.15.78:0/921300588' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-16T01:00:47.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:47 smithi078 ceph-mon[30575]: from='client.? 172.21.15.78:0/921300588' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-16T01:00:48.594 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:48 smithi078 ceph-mon[30575]: from='client.? 172.21.15.78:0/3066634403' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.917+0000 7f1c45016640 1 Processor -- start 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.917+0000 7f1c45016640 1 -- start start 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.917+0000 7f1c45016640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.917+0000 7f1c45016640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c40107010 con 0x7f1c40106640 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.918+0000 7f1c3ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.918+0000 7f1c3ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37612/0 (socket says 172.21.15.78:37612) 2024-09-16T01:00:48.601 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.918+0000 7f1c3ed9d640 1 -- 172.21.15.78:0/1530011034 learned_addr learned my addr 172.21.15.78:0/1530011034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.918+0000 7f1c3ed9d640 1 -- 172.21.15.78:0/1530011034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c401077f0 con 0x7f1c40106640 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.918+0000 7f1c3ed9d640 1 --2- 172.21.15.78:0/1530011034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009820 tx=0x7f1c3002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abac9992f27d5ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c3e59c640 1 -- 172.21.15.78:0/1530011034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c30037070 con 0x7f1c40106640 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c3e59c640 1 -- 172.21.15.78:0/1530011034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c30032900 con 0x7f1c40106640 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c3e59c640 1 -- 172.21.15.78:0/1530011034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c30030850 con 0x7f1c40106640 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 -- 172.21.15.78:0/1530011034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 msgr2=0x7f1c40106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 --2- 172.21.15.78:0/1530011034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009820 tx=0x7f1c3002a270 comp rx=0 tx=0).stop 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 -- 172.21.15.78:0/1530011034 shutdown_connections 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 --2- 172.21.15.78:0/1530011034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c40106a40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 -- 172.21.15.78:0/1530011034 >> 172.21.15.78:0/1530011034 conn(0x7f1c40101db0 msgr2=0x7f1c40104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 -- 172.21.15.78:0/1530011034 shutdown_connections 2024-09-16T01:00:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.919+0000 7f1c45016640 1 -- 172.21.15.78:0/1530011034 wait complete. 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c45016640 1 Processor -- start 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c45016640 1 -- start start 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c45016640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c45016640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4010aba0 con 0x7f1c40106640 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c3ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c3ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37616/0 (socket says 172.21.15.78:37616) 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.920+0000 7f1c3ed9d640 1 -- 172.21.15.78:0/3066634403 learned_addr learned my addr 172.21.15.78:0/3066634403 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3ed9d640 1 -- 172.21.15.78:0/3066634403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4019e4f0 con 0x7f1c40106640 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3ed9d640 1 --2- 172.21.15.78:0/3066634403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097f0 tx=0x7f1c30030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c30042070 con 0x7f1c40106640 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4019e6c0 con 0x7f1c40106640 2024-09-16T01:00:48.603 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c30032ca0 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c30037040 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4019ea60 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1c30045070 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.921+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1c30047a40 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.922+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c4010ad20 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.924+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1c3003da60 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.982+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1c4019a2d0 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c3cd99640 1 -- 172.21.15.78:0/3066634403 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f1c3003d430 con 0x7f1c40106640 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 msgr2=0x7f1c4007ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c45016640 1 --2- 172.21.15.78:0/3066634403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097f0 tx=0x7f1c30030ec0 comp rx=0 tx=0).stop 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 shutdown_connections 2024-09-16T01:00:48.604 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c45016640 1 --2- 172.21.15.78:0/3066634403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c40106640 0x7f1c4007ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:48.605 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.983+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 >> 172.21.15.78:0/3066634403 conn(0x7f1c40101db0 msgr2=0x7f1c40102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:48.605 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.984+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 shutdown_connections 2024-09-16T01:00:48.605 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:47.984+0000 7f1c45016640 1 -- 172.21.15.78:0/3066634403 wait complete. 2024-09-16T01:00:48.605 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2024-09-16T01:00:48.889 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:48 smithi078 systemd[1]: Stopping Ceph mon.a for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:00:49.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a[30571]: 2024-09-16T01:00:48.888+0000 7fccfdd4a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:00:49.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a[30571]: 2024-09-16T01:00:48.888+0000 7fccfdd4a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-16T01:00:49.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:48 smithi078 podman[30817]: 2024-09-16 01:00:48.960262176 +0000 UTC m=+0.195719429 container died 949bcc76e299b71deb6be7f74c547cf269df2d3d2eb8facd35546ae213a98d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-16T01:00:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 podman[30817]: 2024-09-16 01:00:49.346656389 +0000 UTC m=+0.582113625 container cleanup 949bcc76e299b71deb6be7f74c547cf269df2d3d2eb8facd35546ae213a98d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-16T01:00:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 bash[30817]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a 2024-09-16T01:00:49.906 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 podman[30830]: 2024-09-16 01:00:49.580999536 +0000 UTC m=+0.619041884 container remove 949bcc76e299b71deb6be7f74c547cf269df2d3d2eb8facd35546ae213a98d48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T01:00:49.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-16T01:00:49.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 systemd[1]: Stopped Ceph mon.a for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:00:49.907 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 systemd[1]: Starting Ceph mon.a for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:00:50.145 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-16T01:00:50.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 podman[30918]: 2024-09-16 01:00:49.906561109 +0000 UTC m=+0.093876826 container create edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T01:00:50.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:49 smithi078 podman[30918]: 2024-09-16 01:00:49.828292745 +0000 UTC m=+0.015608475 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:00:50.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 podman[30918]: 2024-09-16 01:00:50.075353824 +0000 UTC m=+0.262669550 container init edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T01:00:50.164 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 podman[30918]: 2024-09-16 01:00:50.078367869 +0000 UTC m=+0.265683596 container start edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: set uid:gid to 167:167 (ceph:ceph) 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: pidfile_write: ignore empty --pid-file 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: load: jerasure load: lrc 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: RocksDB version: 7.9.2 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Git sha 0 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: DB SUMMARY 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: DB Session ID: B99ZPSRGVB3PFXCNF6TP 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: CURRENT file: CURRENT 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: IDENTITY file: IDENTITY 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-16T01:00:50.165 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-16T01:00:50.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106382 ; 2024-09-16T01:00:50.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.error_if_exists: 0 2024-09-16T01:00:50.166 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.create_if_missing: 0 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.paranoid_checks: 1 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.env: 0x556e49c6e940 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.fs: PosixFileSystem 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.info_log: 0x556e4a7cf6e0 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_file_opening_threads: 16 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.statistics: (nil) 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.use_fsync: 0 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_log_file_size: 0 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-16T01:00:50.167 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.keep_log_file_num: 1000 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.recycle_log_file_num: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_fallocate: 1 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_mmap_reads: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_mmap_writes: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.use_direct_reads: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.create_missing_column_families: 0 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.db_log_dir: 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.wal_dir: 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-16T01:00:50.168 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.advise_random_on_open: 1 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.db_write_buffer_size: 0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.write_buffer_manager: 0x556e4a7dedc0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.rate_limiter: (nil) 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-16T01:00:50.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.wal_recovery_mode: 2 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enable_thread_tracking: 0 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enable_pipelined_write: 0 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.unordered_write: 0 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.row_cache: None 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.wal_filter: None 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_ingest_behind: 0 2024-09-16T01:00:50.170 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.two_write_queues: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.manual_wal_flush: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.wal_compression: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.atomic_flush: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.log_readahead_size: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.best_efforts_recovery: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.allow_data_in_errors: 0 2024-09-16T01:00:50.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.db_host_id: __hostname__ 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_background_jobs: 2 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_background_compactions: -1 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_subcompactions: 1 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_total_wal_size: 0 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-16T01:00:50.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_open_files: -1 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bytes_per_sync: 0 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_readahead_size: 0 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_background_flushes: -1 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Compression algorithms supported: 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kZSTD supported: 0 2024-09-16T01:00:50.173 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kXpressCompression supported: 0 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kBZip2Compression supported: 0 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kLZ4Compression supported: 1 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kZlibCompression supported: 1 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kLZ4HCCompression supported: 1 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: kSnappyCompression supported: 1 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.merge_operator: 2024-09-16T01:00:50.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_filter: None 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_filter_factory: None 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.sst_partitioner_factory: None 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556e4a7cf320) 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-09-16T01:00:50.175 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x556e4a7c31f0 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-09-16T01:00:50.176 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-09-16T01:00:50.177 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.write_buffer_size: 33554432 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_write_buffer_number: 2 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression: NoCompression 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression: Disabled 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.prefix_extractor: nullptr 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.num_levels: 7 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-16T01:00:50.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.level: 32767 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.strategy: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-16T01:00:50.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-16T01:00:50.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-16T01:00:50.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-16T01:00:50.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.enabled: false 2024-09-16T01:00:50.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-16T01:00:50.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.target_file_size_base: 67108864 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-16T01:00:50.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.arena_block_size: 1048576 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-16T01:00:50.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.disable_auto_compactions: 0 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-16T01:00:50.183 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.inplace_update_support: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.bloom_locality: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.max_successive_merges: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-16T01:00:50.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.paranoid_file_checks: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.force_consistency_checks: 1 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.report_bg_io_stats: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.ttl: 2592000 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enable_blob_files: false 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.min_blob_size: 0 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_file_size: 268435456 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-16T01:00:50.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.blob_file_starting_level: 0 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ece98965-49fb-4589-9f37-3454a97c24a5 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726448450103657, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726448450104735, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101402, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 99278, "index_size": 316, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14564, "raw_average_key_size": 50, "raw_value_size": 91353, "raw_average_value_size": 313, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726448450, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ece98965-49fb-4589-9f37-3454a97c24a5", "db_session_id": "B99ZPSRGVB3PFXCNF6TP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726448450104837, "job": 1, "event": "recovery_finished"} 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-16T01:00:50.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556e4a7f0e00 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: DB pointer 0x556e4a8ec000 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.187 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 100.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 100.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-16T01:00:50.188 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 30.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 30.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x556e4a7c31f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.189 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???) e1 preinit fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).mds e1 new map 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).mds e1 print_map 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: btime 2024-09-16T01:00:45:346505+0000 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-16T01:00:50.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 bash[30918]: edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 systemd[1]: Started Ceph mon.a for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: monmap epoch 1 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: last_changed 2024-09-16T01:00:42.542629+0000 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: created 2024-09-16T01:00:42.542629+0000 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: min_mon_release 19 (squid) 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: election_strategy: 1 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-16T01:00:50.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: fsmap 2024-09-16T01:00:50.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: osdmap e1: 0 total, 0 up, 0 in 2024-09-16T01:00:50.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:50 smithi078 ceph-mon[30937]: mgrmap e1: no daemons active 2024-09-16T01:00:51.184 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.507+0000 7fd5c86c9640 1 Processor -- start 2024-09-16T01:00:51.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.507+0000 7fd5c86c9640 1 -- start start 2024-09-16T01:00:51.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.507+0000 7fd5c86c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:51.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.507+0000 7fd5c86c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0108170 con 0x7fd5c01077a0 2024-09-16T01:00:51.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.508+0000 7fd5c6465640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.508+0000 7fd5c6465640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37620/0 (socket says 172.21.15.78:37620) 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.508+0000 7fd5c6465640 1 -- 172.21.15.78:0/1737235986 learned_addr learned my addr 172.21.15.78:0/1737235986 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.508+0000 7fd5c6465640 1 -- 172.21.15.78:0/1737235986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c0108950 con 0x7fd5c01077a0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.510+0000 7fd5c6465640 1 --2- 172.21.15.78:0/1737235986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009820 tx=0x7fd5bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2985088193cf67d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.510+0000 7fd5c5463640 1 -- 172.21.15.78:0/1737235986 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5bc037070 con 0x7fd5c01077a0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.510+0000 7fd5c5463640 1 -- 172.21.15.78:0/1737235986 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5bc032900 con 0x7fd5c01077a0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.510+0000 7fd5c5463640 1 -- 172.21.15.78:0/1737235986 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5bc030850 con 0x7fd5c01077a0 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- 172.21.15.78:0/1737235986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 msgr2=0x7fd5c0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 --2- 172.21.15.78:0/1737235986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009820 tx=0x7fd5bc02a270 comp rx=0 tx=0).stop 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- 172.21.15.78:0/1737235986 shutdown_connections 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 --2- 172.21.15.78:0/1737235986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c0107ba0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- 172.21.15.78:0/1737235986 >> 172.21.15.78:0/1737235986 conn(0x7fd5c0102fd0 msgr2=0x7fd5c01053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:51.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- 172.21.15.78:0/1737235986 shutdown_connections 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- 172.21.15.78:0/1737235986 wait complete. 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 Processor -- start 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.511+0000 7fd5c86c9640 1 -- start start 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c86c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c86c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c010bd00 con 0x7fd5c01077a0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c6465640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c6465640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37626/0 (socket says 172.21.15.78:37626) 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c6465640 1 -- 172.21.15.78:0/3524233227 learned_addr learned my addr 172.21.15.78:0/3524233227 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c6465640 1 -- 172.21.15.78:0/3524233227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c019bc80 con 0x7fd5c01077a0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c6465640 1 --2- 172.21.15.78:0/3524233227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009940 tx=0x7fd5bc032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5bc042070 con 0x7fd5c01077a0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5bc0308c0 con 0x7fd5c01077a0 2024-09-16T01:00:51.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c019bf10 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5bc037040 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.512+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c019ec00 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.513+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5bc045070 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.513+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd5bc047b70 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.513+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c010be80 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.516+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fd5bc0327b0 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.571+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fd5c019f1f0 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.574+0000 7fd5b37fe640 1 -- 172.21.15.78:0/3524233227 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fd5bc03c760 con 0x7fd5c01077a0 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 msgr2=0x7fd5c019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 --2- 172.21.15.78:0/3524233227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009940 tx=0x7fd5bc032ce0 comp rx=0 tx=0).stop 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 shutdown_connections 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 --2- 172.21.15.78:0/3524233227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c01077a0 0x7fd5c019b740 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 >> 172.21.15.78:0/3524233227 conn(0x7fd5c0102fd0 msgr2=0x7fd5c0103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:51.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 shutdown_connections 2024-09-16T01:00:51.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:50.575+0000 7fd5c86c9640 1 -- 172.21.15.78:0/3524233227 wait complete. 2024-09-16T01:00:51.192 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-16T01:00:51.215 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:00:51.215 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2024-09-16T01:00:51.215 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-16T01:00:51.215 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-16T01:00:51.446 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x 2024-09-16T01:00:51.447 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-16T01:00:51.597 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968.target.wants/ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-fa4731de-73c6-11ef-bceb-c7b262605968@.service. 2024-09-16T01:00:51.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:51 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3524233227' entity='client.admin' 2024-09-16T01:00:51.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:51 smithi078 systemd[1]: Starting Ceph mgr.x for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:00:52.127 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:51 smithi078 podman[31151]: 2024-09-16 01:00:51.729562557 +0000 UTC m=+0.015741951 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:00:52.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:51 smithi078 podman[31151]: 2024-09-16 01:00:51.833454243 +0000 UTC m=+0.119633622 container create 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-16T01:00:52.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 podman[31151]: 2024-09-16 01:00:52.043959618 +0000 UTC m=+0.330138999 container init 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:00:52.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 podman[31151]: 2024-09-16 01:00:52.046951848 +0000 UTC m=+0.333131229 container start 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-16T01:00:52.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 bash[31151]: 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 2024-09-16T01:00:52.142 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T01:00:52.142 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-16T01:00:52.149 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T01:00:52.149 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:00:52.149 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-16T01:00:52.150 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-09-16T01:00:52.158 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T01:00:52.158 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2024-09-16T01:00:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2024-09-16T01:00:52.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 systemd[1]: Started Ceph mgr.x for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:00:52.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.167+0000 7f4ca3067040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:00:52.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.228+0000 7f4ca3067040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:00:52.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.378+0000 7f4ca3067040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:00:52.629 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.448+0000 7f4ca3067040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:00:52.630 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.600+0000 7f4ca3067040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:00:53.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2090260823' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T01:00:53.028 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:52.752+0000 7f4ca3067040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T01:00:53.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:53.298 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T01:00:53.299 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T01:00:53.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T01:00:45:346505+0000", 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T01:00:45.347143+0000", 2024-09-16T01:00:53.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.627+0000 7fdd62a50640 1 Processor -- start 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.627+0000 7fdd62a50640 1 -- start start 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd62a50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd62a50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c07c690 con 0x7fdd5c108250 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37668/0 (socket says 172.21.15.78:37668) 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd5bfff640 1 -- 172.21.15.78:0/3305476414 learned_addr learned my addr 172.21.15.78:0/3305476414 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.628+0000 7fdd5bfff640 1 -- 172.21.15.78:0/3305476414 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c10ab80 con 0x7fdd5c108250 2024-09-16T01:00:53.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.629+0000 7fdd5bfff640 1 --2- 172.21.15.78:0/3305476414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009820 tx=0x7fdd4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43142e15376b5c79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.630+0000 7fdd5b7fe640 1 -- 172.21.15.78:0/3305476414 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd4c037070 con 0x7fdd5c108250 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.630+0000 7fdd5b7fe640 1 -- 172.21.15.78:0/3305476414 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdd4c032900 con 0x7fdd5c108250 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.630+0000 7fdd5b7fe640 1 -- 172.21.15.78:0/3305476414 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd4c030850 con 0x7fdd5c108250 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.630+0000 7fdd62a50640 1 -- 172.21.15.78:0/3305476414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 msgr2=0x7fdd5c10a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.630+0000 7fdd62a50640 1 --2- 172.21.15.78:0/3305476414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009820 tx=0x7fdd4c02a270 comp rx=0 tx=0).stop 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- 172.21.15.78:0/3305476414 shutdown_connections 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 --2- 172.21.15.78:0/3305476414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c10a640 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- 172.21.15.78:0/3305476414 >> 172.21.15.78:0/3305476414 conn(0x7fdd5c101e50 msgr2=0x7fdd5c104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- 172.21.15.78:0/3305476414 shutdown_connections 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- 172.21.15.78:0/3305476414 wait complete. 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 Processor -- start 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- start start 2024-09-16T01:00:53.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.631+0000 7fdd62a50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c10bb60 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37678/0 (socket says 172.21.15.78:37678) 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd5bfff640 1 -- 172.21.15.78:0/2090260823 learned_addr learned my addr 172.21.15.78:0/2090260823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd5bfff640 1 -- 172.21.15.78:0/2090260823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c19bcb0 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd5bfff640 1 --2- 172.21.15.78:0/2090260823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0097f0 tx=0x7fdd4c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd4c037040 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdd4c040070 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5c19bf40 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd4c03a390 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.632+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5c19ec30 con 0x7fdd5c108250 2024-09-16T01:00:53.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.633+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdd4c03a600 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.633+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdd4c0459a0 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.633+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5c1086b0 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.635+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fdd4c045460 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.697+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdd5c19ee00 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.697+0000 7fdd59ffb640 1 -- 172.21.15.78:0/2090260823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7fdd4c03b430 con 0x7fdd5c108250 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 msgr2=0x7fdd5c19b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 --2- 172.21.15.78:0/2090260823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0097f0 tx=0x7fdd4c030d80 comp rx=0 tx=0).stop 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 shutdown_connections 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 --2- 172.21.15.78:0/2090260823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5c108250 0x7fdd5c19b770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 >> 172.21.15.78:0/2090260823 conn(0x7fdd5c101e50 msgr2=0x7fdd5c102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 shutdown_connections 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:52.698+0000 7fdd62a50640 1 -- 172.21.15.78:0/2090260823 wait complete. 2024-09-16T01:00:53.313 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2024-09-16T01:00:53.330 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:53.074+0000 7f4ca3067040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:00:53.330 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:53.136+0000 7f4ca3067040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:00:53.330 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:53.323+0000 7f4ca3067040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:00:54.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:54.199+0000 7f4ca3067040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:00:54.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:54.265+0000 7f4ca3067040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:00:55.235 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:54.977+0000 7f4ca3067040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:00:55.235 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.092+0000 7f4ca3067040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:00:55.235 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.234+0000 7f4ca3067040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:00:55.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.294+0000 7f4ca3067040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:00:55.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.448+0000 7f4ca3067040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:00:55.521 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.517+0000 7f4ca3067040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:00:55.778 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.578+0000 7f4ca3067040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:00:55.778 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.741+0000 7f4ca3067040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:00:56.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3458318714' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T01:00:56.092 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.813+0000 7f4ca3067040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:00:56.092 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.884+0000 7f4ca3067040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:00:56.093 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:55.967+0000 7f4ca3067040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:00:56.093 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:56.032+0000 7f4ca3067040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:00:56.093 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:56.091+0000 7f4ca3067040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T01:00:56.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:56.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T01:00:56.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T01:00:45:346505+0000", 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T01:00:56.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T01:00:45.347143+0000", 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.720+0000 7f0bff4cd640 1 Processor -- start 2024-09-16T01:00:56.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bff4cd640 1 -- start start 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bff4cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bff4cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf807be30 con 0x7f0bf81081b0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bfd269640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bfd269640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45994/0 (socket says 172.21.15.78:45994) 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bfd269640 1 -- 172.21.15.78:0/466436724 learned_addr learned my addr 172.21.15.78:0/466436724 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.721+0000 7f0bfd269640 1 -- 172.21.15.78:0/466436724 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf810aae0 con 0x7f0bf81081b0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.722+0000 7f0bfd269640 1 --2- 172.21.15.78:0/466436724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009820 tx=0x7f0bf402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc089415be45e505 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.722+0000 7f0beffff640 1 -- 172.21.15.78:0/466436724 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4037070 con 0x7f0bf81081b0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.722+0000 7f0beffff640 1 -- 172.21.15.78:0/466436724 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0bf4032900 con 0x7f0bf81081b0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.722+0000 7f0beffff640 1 -- 172.21.15.78:0/466436724 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4030850 con 0x7f0bf81081b0 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 -- 172.21.15.78:0/466436724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 msgr2=0x7f0bf810a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 --2- 172.21.15.78:0/466436724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009820 tx=0x7f0bf402a270 comp rx=0 tx=0).stop 2024-09-16T01:00:56.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 -- 172.21.15.78:0/466436724 shutdown_connections 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 --2- 172.21.15.78:0/466436724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf810a5a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 -- 172.21.15.78:0/466436724 >> 172.21.15.78:0/466436724 conn(0x7f0bf8101db0 msgr2=0x7f0bf8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 -- 172.21.15.78:0/466436724 shutdown_connections 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 -- 172.21.15.78:0/466436724 wait complete. 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.723+0000 7f0bff4cd640 1 Processor -- start 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bff4cd640 1 -- start start 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bff4cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bff4cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf810bb20 con 0x7f0bf81081b0 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bfd269640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bfd269640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46010/0 (socket says 172.21.15.78:46010) 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bfd269640 1 -- 172.21.15.78:0/3458318714 learned_addr learned my addr 172.21.15.78:0/3458318714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bfd269640 1 -- 172.21.15.78:0/3458318714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf819bc90 con 0x7f0bf81081b0 2024-09-16T01:00:56.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bfd269640 1 --2- 172.21.15.78:0/3458318714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40097f0 tx=0x7f0bf4030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4042070 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0bf4032ca0 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf819bf20 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.724+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bf4037040 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.725+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf819ec10 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.725+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0bf4045070 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.725+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0bf40463b0 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.725+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf8108610 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.728+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0bf403db10 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.789+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0bf819f180 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.790+0000 7f0bee7fc640 1 -- 172.21.15.78:0/3458318714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f0bf4047440 con 0x7f0bf81081b0 2024-09-16T01:00:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 msgr2=0x7f0bf819b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 --2- 172.21.15.78:0/3458318714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40097f0 tx=0x7f0bf4030ec0 comp rx=0 tx=0).stop 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 shutdown_connections 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 --2- 172.21.15.78:0/3458318714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf81081b0 0x7f0bf819b750 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 >> 172.21.15.78:0/3458318714 conn(0x7f0bf8101db0 msgr2=0x7f0bf8102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 shutdown_connections 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:55.791+0000 7f0bff4cd640 1 -- 172.21.15.78:0/3458318714 wait complete. 2024-09-16T01:00:56.453 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2024-09-16T01:00:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: Activating manager daemon x 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: mgrmap e2: x(active, starting, since 0.00322769s) 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: Manager daemon x is now available 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:00:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' 2024-09-16T01:00:57.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-mon[30937]: from='mgr.14100 172.21.15.78:0/2727961136' entity='mgr.x' 2024-09-16T01:00:57.280 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:00:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:00:56.878+0000 7f4ca3067040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:00:59.074 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:58 smithi078 ceph-mon[30937]: mgrmap e3: x(active, since 1.00677s) 2024-09-16T01:00:59.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:00:59.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:00:59.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:00:59.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-09-16T01:00:59.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-16T01:00:59.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-16T01:00:59.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-16T01:00:59.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-16T01:00:59.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-16T01:00:59.670 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-16T01:00:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "btime": "2024-09-16T01:00:45:346505+0000", 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-16T01:00:59.672 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-09-16T01:00:45.347143+0000", 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc283f9f640 1 Processor -- start 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc283f9f640 1 -- start start 2024-09-16T01:00:59.673 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc283f9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc283f9f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc27c07d080 con 0x7fc27c07e690 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc281d3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc281d3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46062/0 (socket says 172.21.15.78:46062) 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc281d3b640 1 -- 172.21.15.78:0/3087133281 learned_addr learned my addr 172.21.15.78:0/3087133281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.822+0000 7fc281d3b640 1 -- 172.21.15.78:0/3087133281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27c07d200 con 0x7fc27c07e690 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.823+0000 7fc281d3b640 1 --2- 172.21.15.78:0/3087133281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc270009820 tx=0x7fc27002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc315c3fa8a5408 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.823+0000 7fc280d39640 1 -- 172.21.15.78:0/3087133281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc270037070 con 0x7fc27c07e690 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.823+0000 7fc280d39640 1 -- 172.21.15.78:0/3087133281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc270032900 con 0x7fc27c07e690 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.823+0000 7fc280d39640 1 -- 172.21.15.78:0/3087133281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc270030850 con 0x7fc27c07e690 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 -- 172.21.15.78:0/3087133281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 msgr2=0x7fc27c07cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3087133281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc270009820 tx=0x7fc27002a270 comp rx=0 tx=0).stop 2024-09-16T01:00:59.674 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 -- 172.21.15.78:0/3087133281 shutdown_connections 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3087133281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c07cb40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 -- 172.21.15.78:0/3087133281 >> 172.21.15.78:0/3087133281 conn(0x7fc27c102fa0 msgr2=0x7fc27c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 -- 172.21.15.78:0/3087133281 shutdown_connections 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.824+0000 7fc283f9f640 1 -- 172.21.15.78:0/3087133281 wait complete. 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc283f9f640 1 Processor -- start 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc283f9f640 1 -- start start 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc283f9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc283f9f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc27c10ddc0 con 0x7fc27c07e690 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc281d3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc281d3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46066/0 (socket says 172.21.15.78:46066) 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc281d3b640 1 -- 172.21.15.78:0/3654895013 learned_addr learned my addr 172.21.15.78:0/3654895013 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc281d3b640 1 -- 172.21.15.78:0/3654895013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27c19de50 con 0x7fc27c07e690 2024-09-16T01:00:59.675 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.825+0000 7fc281d3b640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc2700097f0 tx=0x7fc270030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.826+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc270037040 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.826+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc270040070 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.826+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27c19e0e0 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.826+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc27003a390 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.826+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27c1a0dd0 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.827+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7fc270032900 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.827+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc27c07cbc0 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.827+0000 7fc26effd640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 0x7fc26803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.827+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc27006c160 con 0x7fc27c07e690 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.827+0000 7fc28153a640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 0x7fc26803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.828+0000 7fc28153a640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 0x7fc26803b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc2780098c0 tx=0x7fc278006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:00:59.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.832+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc27003a5e0 con 0x7fc27c07e690 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:58.887+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fc270032bf0 con 0x7fc27c07e690 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.062+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc27c1a0fa0 con 0x7fc27c07e690 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.062+0000 7fc26effd640 1 -- 172.21.15.78:0/3654895013 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7fc270062f50 con 0x7fc27c07e690 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.064+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 msgr2=0x7fc26803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.064+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 0x7fc26803b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc2780098c0 tx=0x7fc278006db0 comp rx=0 tx=0).stop 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.064+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 msgr2=0x7fc27c19d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.064+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc2700097f0 tx=0x7fc270030d80 comp rx=0 tx=0).stop 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 shutdown_connections 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7fc2680390a0 0x7fc26803b560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 --2- 172.21.15.78:0/3654895013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc27c07e690 0x7fc27c19d910 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 >> 172.21.15.78:0/3654895013 conn(0x7fc27c102fa0 msgr2=0x7fc27c103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 shutdown_connections 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:00:59.065+0000 7fc283f9f640 1 -- 172.21.15.78:0/3654895013 wait complete. 2024-09-16T01:00:59.677 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2024-09-16T01:01:00.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:59 smithi078 ceph-mon[30937]: mgrmap e4: x(active, since 2s) 2024-09-16T01:01:00.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:00:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3654895013' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-16T01:01:01.273 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:00 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3236856325' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-16T01:01:04.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:01:04.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.067+0000 7f0465a66640 1 Processor -- start 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.067+0000 7f0465a66640 1 -- start start 2024-09-16T01:01:04.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.067+0000 7f0465a66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.067+0000 7f0465a66640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0460107010 con 0x7f0460106640 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.068+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.068+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46068/0 (socket says 172.21.15.78:46068) 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.068+0000 7f045f7fe640 1 -- 172.21.15.78:0/1118920843 learned_addr learned my addr 172.21.15.78:0/1118920843 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.068+0000 7f045f7fe640 1 -- 172.21.15.78:0/1118920843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04601077f0 con 0x7f0460106640 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f045f7fe640 1 --2- 172.21.15.78:0/1118920843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0450009820 tx=0x7f045002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c3063d3693bbda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f045effd640 1 -- 172.21.15.78:0/1118920843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0450037070 con 0x7f0460106640 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f045effd640 1 -- 172.21.15.78:0/1118920843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0450032900 con 0x7f0460106640 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f045effd640 1 -- 172.21.15.78:0/1118920843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0450030850 con 0x7f0460106640 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 -- 172.21.15.78:0/1118920843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 msgr2=0x7f0460106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 --2- 172.21.15.78:0/1118920843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0450009820 tx=0x7f045002a270 comp rx=0 tx=0).stop 2024-09-16T01:01:04.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 -- 172.21.15.78:0/1118920843 shutdown_connections 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 --2- 172.21.15.78:0/1118920843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f0460106a40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 -- 172.21.15.78:0/1118920843 >> 172.21.15.78:0/1118920843 conn(0x7f0460101db0 msgr2=0x7f0460104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 -- 172.21.15.78:0/1118920843 shutdown_connections 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.069+0000 7f0465a66640 1 -- 172.21.15.78:0/1118920843 wait complete. 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f0465a66640 1 Processor -- start 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f0465a66640 1 -- start start 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f0465a66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f0465a66640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f046010aba0 con 0x7f0460106640 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f045f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46080/0 (socket says 172.21.15.78:46080) 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.070+0000 7f045f7fe640 1 -- 172.21.15.78:0/3236856325 learned_addr learned my addr 172.21.15.78:0/3236856325 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:04.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f045f7fe640 1 -- 172.21.15.78:0/3236856325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046019bd70 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f045f7fe640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0450009940 tx=0x7f04500326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0450042070 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0450032be0 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046019c000 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0450037040 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.071+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046019ecf0 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f0450045070 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f045d7fa640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 0x7f044c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f045006c100 con 0x7f0460106640 2024-09-16T01:01:04.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f046010ad20 con 0x7f0460106640 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f0456ffd640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 0x7f044c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.072+0000 7f0456ffd640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 0x7f044c03b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04480098c0 tx=0x7f0448006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.077+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0450032d90 con 0x7f0460106640 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.244+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0460106ac0 con 0x7f0460106640 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.245+0000 7f045d7fa640 1 -- 172.21.15.78:0/3236856325 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7f045002ed90 con 0x7f0460106640 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 msgr2=0x7f044c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 0x7f044c03b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04480098c0 tx=0x7f0448006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 msgr2=0x7f046019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0450009940 tx=0x7f04500326e0 comp rx=0 tx=0).stop 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 shutdown_connections 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f044c039160 0x7f044c03b620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:04.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 --2- 172.21.15.78:0/3236856325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0460106640 0x7f046019b830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:04.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 >> 172.21.15.78:0/3236856325 conn(0x7f0460101db0 msgr2=0x7f0460102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:04.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 shutdown_connections 2024-09-16T01:01:04.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:00.248+0000 7f0465a66640 1 -- 172.21.15.78:0/3236856325 wait complete. 2024-09-16T01:01:04.630 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2024-09-16T01:01:08.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/39745438' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-16T01:01:08.214 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setuser ceph since I am not root 2024-09-16T01:01:08.214 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setgroup ceph since I am not root 2024-09-16T01:01:08.215 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.153+0000 7f34f271d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:01:08.215 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.212+0000 7f34f271d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:01:08.489 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.357+0000 7f34f271d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:01:08.489 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.423+0000 7f34f271d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:01:08.778 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.569+0000 7f34f271d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:01:08.778 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:08.709+0000 7f34f271d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:01:09.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:09 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/39745438' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-16T01:01:09.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:09 smithi078 ceph-mon[30937]: mgrmap e5: x(active, since 11s) 2024-09-16T01:01:09.266 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:09.021+0000 7f34f271d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:01:09.266 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:09.081+0000 7f34f271d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:01:09.266 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:09.263+0000 7f34f271d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:01:10.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:10 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:10.144+0000 7f34f271d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:01:10.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:10 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:10.212+0000 7f34f271d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:01:11.192 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:10 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:10.929+0000 7f34f271d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:01:11.192 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.046+0000 7f34f271d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:01:11.473 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.190+0000 7f34f271d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:01:11.474 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.252+0000 7f34f271d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:01:11.474 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.402+0000 7f34f271d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:01:11.474 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.472+0000 7f34f271d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:01:11.756 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.532+0000 7f34f271d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:01:11.756 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.689+0000 7f34f271d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:01:11.756 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.754+0000 7f34f271d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:01:12.022 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.820+0000 7f34f271d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:01:12.022 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.902+0000 7f34f271d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:01:12.022 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:11.962+0000 7f34f271d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:01:12.022 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:12.020+0000 7f34f271d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:01:13.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:12.811+0000 7f34f271d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:01:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: Active manager daemon x restarted 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: Activating manager daemon x 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: osdmap e2: 0 total, 0 up, 0 in 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: mgrmap e6: x(active, starting, since 0.00326158s) 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: Manager daemon x is now available 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:13.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:13.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:13.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:01:13.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:12 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:01:13.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f27880c4640 1 Processor -- start 2024-09-16T01:01:13.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f27880c4640 1 -- start start 2024-09-16T01:01:13.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f27880c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f27880c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f278007c6c0 con 0x7f2780108200 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f2785e60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.727+0000 7f2785e60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34728/0 (socket says 172.21.15.78:34728) 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.728+0000 7f2785e60640 1 -- 172.21.15.78:0/3465179772 learned_addr learned my addr 172.21.15.78:0/3465179772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.728+0000 7f2785e60640 1 -- 172.21.15.78:0/3465179772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278010ab30 con 0x7f2780108200 2024-09-16T01:01:13.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f2785e60640 1 --2- 172.21.15.78:0/3465179772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f277c009820 tx=0x7f277c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e150090095c194e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f2784e5e640 1 -- 172.21.15.78:0/3465179772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f277c037070 con 0x7f2780108200 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f2784e5e640 1 -- 172.21.15.78:0/3465179772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f277c032900 con 0x7f2780108200 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f2784e5e640 1 -- 172.21.15.78:0/3465179772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f277c030850 con 0x7f2780108200 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 -- 172.21.15.78:0/3465179772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 msgr2=0x7f278010a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 --2- 172.21.15.78:0/3465179772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f277c009820 tx=0x7f277c02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 -- 172.21.15.78:0/3465179772 shutdown_connections 2024-09-16T01:01:13.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 --2- 172.21.15.78:0/3465179772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f278010a5f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 -- 172.21.15.78:0/3465179772 >> 172.21.15.78:0/3465179772 conn(0x7f2780101ea0 msgr2=0x7f27801042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 -- 172.21.15.78:0/3465179772 shutdown_connections 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.729+0000 7f27880c4640 1 -- 172.21.15.78:0/3465179772 wait complete. 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f27880c4640 1 Processor -- start 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f27880c4640 1 -- start start 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f27880c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f27880c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f278010bb70 con 0x7f2780108200 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f2785e60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f2785e60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34730/0 (socket says 172.21.15.78:34730) 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f2785e60640 1 -- 172.21.15.78:0/39745438 learned_addr learned my addr 172.21.15.78:0/39745438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.730+0000 7f2785e60640 1 -- 172.21.15.78:0/39745438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780107fc0 con 0x7f2780108200 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f2785e60640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097f0 tx=0x7f277c030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:13.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f277c037040 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27801061b0 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f277c040070 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f277c03a390 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.731+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2780106690 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f277c03a630 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2780108660 con 0x7f2780108200 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f2776ffd640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 0x7f276c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f277c06c530 con 0x7f2780108200 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f278565f640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 0x7f276c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.732+0000 7f278565f640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 0x7f276c03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27700098c0 tx=0x7f2770006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.737+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f277c06c290 con 0x7f2780108200 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.946+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f2780106f70 con 0x7f2780108200 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.997+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f277c043760 con 0x7f2780108200 2024-09-16T01:01:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:07.999+0000 7f2776ffd640 1 -- 172.21.15.78:0/39745438 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f277c03a920 con 0x7f2780108200 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 msgr2=0x7f276c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 0x7f276c03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27700098c0 tx=0x7f2770006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 msgr2=0x7f2780107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097f0 tx=0x7f277c030d80 comp rx=0 tx=0).stop 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 shutdown_connections 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:6800/3073396374,v1:172.21.15.78:6801/3073396374] conn(0x7f276c039160 0x7f276c03b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:13.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 --2- 172.21.15.78:0/39745438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2780108200 0x7f2780107a80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:13.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 >> 172.21.15.78:0/39745438 conn(0x7f2780101ea0 msgr2=0x7f2780102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:13.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 shutdown_connections 2024-09-16T01:01:13.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:08.000+0000 7f27880c4640 1 -- 172.21.15.78:0/39745438 wait complete. 2024-09-16T01:01:14.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:13 smithi078 ceph-mon[30937]: Found migration_current of "None". Setting to last migration. 2024-09-16T01:01:14.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:13 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:14.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:13 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:14.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:13 smithi078 ceph-mon[30937]: mgrmap e7: x(active, since 1.00536s) 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:14.435+0000 7f3459a77640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:01:01:14] ENGINE Error in HTTPServer.serve 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Traceback (most recent call last): 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._connections.run(self.expiration_interval) 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._run(expiration_interval) 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:01:14.525 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s = self.context.wrap_socket( 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: return self.sslsocket_class._create( 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self.do_handshake() 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._sslobj.do_handshake() 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:01:14.526 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 7, 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.987+0000 7f80a1426640 1 Processor -- start 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.987+0000 7f80a1426640 1 -- start start 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.987+0000 7f80a1426640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.987+0000 7f80a1426640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f809c107030 con 0x7f809c106660 2024-09-16T01:01:14.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.988+0000 7f809affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.988+0000 7f809affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34816/0 (socket says 172.21.15.78:34816) 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.988+0000 7f809affd640 1 -- 172.21.15.78:0/1610886093 learned_addr learned my addr 172.21.15.78:0/1610886093 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.988+0000 7f809affd640 1 -- 172.21.15.78:0/1610886093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c107860 con 0x7f809c106660 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f809affd640 1 --2- 172.21.15.78:0/1610886093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f808c009820 tx=0x7f808c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dd0c559d390a01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f8099ffb640 1 -- 172.21.15.78:0/1610886093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f808c037070 con 0x7f809c106660 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f8099ffb640 1 -- 172.21.15.78:0/1610886093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f808c032900 con 0x7f809c106660 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f8099ffb640 1 -- 172.21.15.78:0/1610886093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f808c030850 con 0x7f809c106660 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 -- 172.21.15.78:0/1610886093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 msgr2=0x7f809c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 --2- 172.21.15.78:0/1610886093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f808c009820 tx=0x7f808c02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 -- 172.21.15.78:0/1610886093 shutdown_connections 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 --2- 172.21.15.78:0/1610886093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c106a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:14.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 -- 172.21.15.78:0/1610886093 >> 172.21.15.78:0/1610886093 conn(0x7f809c101e50 msgr2=0x7f809c104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 -- 172.21.15.78:0/1610886093 shutdown_connections 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.989+0000 7f80a1426640 1 -- 172.21.15.78:0/1610886093 wait complete. 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f80a1426640 1 Processor -- start 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f80a1426640 1 -- start start 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f80a1426640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f80a1426640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f809c07b9b0 con 0x7f809c106660 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f809affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f809affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34818/0 (socket says 172.21.15.78:34818) 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.990+0000 7f809affd640 1 -- 172.21.15.78:0/755513478 learned_addr learned my addr 172.21.15.78:0/755513478 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f809affd640 1 -- 172.21.15.78:0/755513478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c19bd30 con 0x7f809c106660 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f809affd640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f808c009940 tx=0x7f808c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f808c042070 con 0x7f809c106660 2024-09-16T01:01:14.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f808c0308c0 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809c19bfc0 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f808c037040 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.991+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809c19ec60 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.992+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7f808c045070 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.992+0000 7f8083fff640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 0x7f808403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.992+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809c106a60 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.992+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f808c06c170 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.992+0000 7f809a7fc640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 0x7f808403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.993+0000 7f809a7fc640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 0x7f808403b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f80900098c0 tx=0x7f8090006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:13.997+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f808c03cd90 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.206+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f809c02da10 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.207+0000 7f8083fff640 1 -- 172.21.15.78:0/755513478 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v7) ==== 56+0+88 (secure 0 0 0) 0x7f808c0060e0 con 0x7f809c106660 2024-09-16T01:01:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 msgr2=0x7f808403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 0x7f808403b560 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f80900098c0 tx=0x7f8090006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 msgr2=0x7f809c19b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f808c009940 tx=0x7f808c032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 shutdown_connections 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f80840390a0 0x7f808403b560 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 --2- 172.21.15.78:0/755513478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809c106660 0x7f809c19b7f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 >> 172.21.15.78:0/755513478 conn(0x7f809c101e50 msgr2=0x7f809c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 shutdown_connections 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:14.209+0000 7f80a1426640 1 -- 172.21.15.78:0/755513478 wait complete. 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-09-16T01:01:14.803 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 7... 2024-09-16T01:01:15.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/755513478' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T01:01:15.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:14 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:01:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-16T01:01:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T01:01:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:01:15.856 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e539da640 1 Processor -- start 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e539da640 1 -- start start 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e539da640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e539da640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c108170 con 0x7f8e4c1077a0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e51776640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e51776640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51352/0 (socket says 172.21.15.78:51352) 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.230+0000 7f8e51776640 1 -- 172.21.15.78:0/1930366135 learned_addr learned my addr 172.21.15.78:0/1930366135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.231+0000 7f8e51776640 1 -- 172.21.15.78:0/1930366135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c108950 con 0x7f8e4c1077a0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.231+0000 7f8e51776640 1 --2- 172.21.15.78:0/1930366135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009820 tx=0x7f8e4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d7cd87868f747b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.231+0000 7f8e43fff640 1 -- 172.21.15.78:0/1930366135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e48037070 con 0x7f8e4c1077a0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.231+0000 7f8e43fff640 1 -- 172.21.15.78:0/1930366135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e48032900 con 0x7f8e4c1077a0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.231+0000 7f8e43fff640 1 -- 172.21.15.78:0/1930366135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e48030850 con 0x7f8e4c1077a0 2024-09-16T01:01:15.857 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 -- 172.21.15.78:0/1930366135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 msgr2=0x7f8e4c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 --2- 172.21.15.78:0/1930366135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009820 tx=0x7f8e4802a270 comp rx=0 tx=0).stop 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 -- 172.21.15.78:0/1930366135 shutdown_connections 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 --2- 172.21.15.78:0/1930366135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c107ba0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 -- 172.21.15.78:0/1930366135 >> 172.21.15.78:0/1930366135 conn(0x7f8e4c102fd0 msgr2=0x7f8e4c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 -- 172.21.15.78:0/1930366135 shutdown_connections 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.232+0000 7f8e539da640 1 -- 172.21.15.78:0/1930366135 wait complete. 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e539da640 1 Processor -- start 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e539da640 1 -- start start 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e539da640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e539da640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10bd00 con 0x7f8e4c1077a0 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e51776640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e51776640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51364/0 (socket says 172.21.15.78:51364) 2024-09-16T01:01:15.858 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e51776640 1 -- 172.21.15.78:0/3892489125 learned_addr learned my addr 172.21.15.78:0/3892489125 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e51776640 1 -- 172.21.15.78:0/3892489125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c19bc80 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e51776640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009940 tx=0x7f8e48032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e48042070 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c19bf10 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e480308c0 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.233+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e48037040 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.234+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c19ec00 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.234+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f8e48045070 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.235+0000 7f8e427fc640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 0x7f8e3803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.235+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e3803bd80 con 0x7f8e380391b0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.235+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8e4806c320 con 0x7f8e4c1077a0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.235+0000 7f8e50f75640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 0x7f8e3803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:15.859 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.235+0000 7f8e50f75640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 0x7f8e3803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0098c0 tx=0x7f8e3c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.236+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f8e3803bd80 con 0x7f8e380391b0 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8e4c10be80 con 0x7f8e380391b0 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e427fc640 1 -- 172.21.15.78:0/3892489125 <== mgr.14118 v2:172.21.15.78:6800/2425087015 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f8e4c10be80 con 0x7f8e380391b0 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 msgr2=0x7f8e3803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e539da640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 0x7f8e3803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c0098c0 tx=0x7f8e3c006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 msgr2=0x7f8e4c19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.242+0000 7f8e539da640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009940 tx=0x7f8e48032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 shutdown_connections 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f8e380391b0 0x7f8e3803b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 --2- 172.21.15.78:0/3892489125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e4c1077a0 0x7f8e4c19b740 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:15.860 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 >> 172.21.15.78:0/3892489125 conn(0x7f8e4c102fd0 msgr2=0x7f8e4c103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:15.861 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 shutdown_connections 2024-09-16T01:01:15.861 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:15.243+0000 7f8e539da640 1 -- 172.21.15.78:0/3892489125 wait complete. 2024-09-16T01:01:15.861 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 7 is available 2024-09-16T01:01:15.861 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2024-09-16T01:01:16.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:14] ENGINE Bus STARTING 2024-09-16T01:01:16.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:14] ENGINE Serving on http://172.21.15.78:8765 2024-09-16T01:01:16.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:14] ENGINE Serving on https://172.21.15.78:7150 2024-09-16T01:01:16.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:14] ENGINE Bus STARTED 2024-09-16T01:01:16.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:14] ENGINE Error in HTTPServer.serve 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:01:16.210 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:01:16.211 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:15 smithi078 ceph-mon[30937]: mgrmap e8: x(active, since 2s) 2024-09-16T01:01:17.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:16 smithi078 ceph-mon[30937]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T01:01:17.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:16 smithi078 ceph-mon[30937]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T01:01:17.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:16 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:17.146 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:16 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2aa8a7640 1 Processor -- start 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2aa8a7640 1 -- start start 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2aa8a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2aa8a7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a4108170 con 0x7ff2a41077a0 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51368/0 (socket says 172.21.15.78:51368) 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2a3fff640 1 -- 172.21.15.78:0/1101810877 learned_addr learned my addr 172.21.15.78:0/1101810877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.292+0000 7ff2a3fff640 1 -- 172.21.15.78:0/1101810877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a4108950 con 0x7ff2a41077a0 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.293+0000 7ff2a3fff640 1 --2- 172.21.15.78:0/1101810877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff294009820 tx=0x7ff29402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11b7c602299c632 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.293+0000 7ff2a2ffd640 1 -- 172.21.15.78:0/1101810877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294037070 con 0x7ff2a41077a0 2024-09-16T01:01:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.293+0000 7ff2a2ffd640 1 -- 172.21.15.78:0/1101810877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff294032900 con 0x7ff2a41077a0 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.293+0000 7ff2a2ffd640 1 -- 172.21.15.78:0/1101810877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294030850 con 0x7ff2a41077a0 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/1101810877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 msgr2=0x7ff2a4107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/1101810877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff294009820 tx=0x7ff29402a270 comp rx=0 tx=0).stop 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/1101810877 shutdown_connections 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/1101810877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a4107ba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/1101810877 >> 172.21.15.78:0/1101810877 conn(0x7ff2a4102fd0 msgr2=0x7ff2a41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/1101810877 shutdown_connections 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.294+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/1101810877 wait complete. 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2aa8a7640 1 Processor -- start 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2aa8a7640 1 -- start start 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2aa8a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:17.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2aa8a7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a410bd00 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51376/0 (socket says 172.21.15.78:51376) 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2a3fff640 1 -- 172.21.15.78:0/3106895939 learned_addr learned my addr 172.21.15.78:0/3106895939 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2a3fff640 1 -- 172.21.15.78:0/3106895939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a419bc80 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.295+0000 7ff2a3fff640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff294009940 tx=0x7ff294032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294042070 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2940308c0 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a419bf10 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff294037040 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a419ec00 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.296+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7ff294045070 con 0x7ff2a41077a0 2024-09-16T01:01:17.155 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.297+0000 7ff2a17fa640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 0x7ff28c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.297+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff29406c560 con 0x7ff2a41077a0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.297+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a410be80 con 0x7ff2a41077a0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.297+0000 7ff2a37fe640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 0x7ff28c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.297+0000 7ff2a37fe640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 0x7ff28c03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2980098c0 tx=0x7ff298006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.302+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff29403cd90 con 0x7ff2a41077a0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.481+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7ff2a4104280 con 0x7ff28c0391b0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.486+0000 7ff2a17fa640 1 -- 172.21.15.78:0/3106895939 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff2a4104280 con 0x7ff28c0391b0 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 msgr2=0x7ff28c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 0x7ff28c03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2980098c0 tx=0x7ff298006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 msgr2=0x7ff2a419b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff294009940 tx=0x7ff294032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:17.156 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 shutdown_connections 2024-09-16T01:01:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7ff28c0391b0 0x7ff28c03b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 --2- 172.21.15.78:0/3106895939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2a41077a0 0x7ff2a419b740 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 >> 172.21.15.78:0/3106895939 conn(0x7ff2a4102fd0 msgr2=0x7ff2a4103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 shutdown_connections 2024-09-16T01:01:17.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:16.488+0000 7ff2aa8a7640 1 -- 172.21.15.78:0/3106895939 wait complete. 2024-09-16T01:01:17.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:17 smithi078 ceph-mon[30937]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-16T01:01:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff96ab640 1 Processor -- start 2024-09-16T01:01:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff96ab640 1 -- start start 2024-09-16T01:01:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff96ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff96ab640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff4108160 con 0x7f1ff4107790 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51390/0 (socket says 172.21.15.78:51390) 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.531+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/3468197648 learned_addr learned my addr 172.21.15.78:0/3468197648 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.532+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/3468197648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff4108940 con 0x7f1ff4107790 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.532+0000 7f1ff2ffd640 1 --2- 172.21.15.78:0/3468197648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009820 tx=0x7f1fe402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af89f64a5703685 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.532+0000 7f1ff27fc640 1 -- 172.21.15.78:0/3468197648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fe4037070 con 0x7f1ff4107790 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff27fc640 1 -- 172.21.15.78:0/3468197648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1fe4032900 con 0x7f1ff4107790 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff27fc640 1 -- 172.21.15.78:0/3468197648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fe4030850 con 0x7f1ff4107790 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 -- 172.21.15.78:0/3468197648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 msgr2=0x7f1ff4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/3468197648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009820 tx=0x7f1fe402a270 comp rx=0 tx=0).stop 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 -- 172.21.15.78:0/3468197648 shutdown_connections 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/3468197648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff4107b90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:18.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 -- 172.21.15.78:0/3468197648 >> 172.21.15.78:0/3468197648 conn(0x7f1ff4102fa0 msgr2=0x7f1ff41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 -- 172.21.15.78:0/3468197648 shutdown_connections 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.533+0000 7f1ff96ab640 1 -- 172.21.15.78:0/3468197648 wait complete. 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff96ab640 1 Processor -- start 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff96ab640 1 -- start start 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff96ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff96ab640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff410bcf0 con 0x7f1ff4107790 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51398/0 (socket says 172.21.15.78:51398) 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/1593187612 learned_addr learned my addr 172.21.15.78:0/1593187612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/1593187612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff419deb0 con 0x7f1ff4107790 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff2ffd640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009940 tx=0x7f1fe4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:18.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fe4042070 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1fe40308c0 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fe4037040 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.534+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff419e140 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.535+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff41a0e30 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.535+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f1fe4045070 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.536+0000 7f1ff0ff9640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 0x7f1fe003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.536+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff410be70 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.536+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1fe406c320 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.536+0000 7f1febfff640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 0x7f1fe003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.536+0000 7f1febfff640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 0x7f1fe003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098c0 tx=0x7f1fdc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.540+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1fe403cd90 con 0x7f1ff4107790 2024-09-16T01:01:18.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.708+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f1ff41041a0 con 0x7f1fe00391b0 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.709+0000 7f1ff0ff9640 1 -- 172.21.15.78:0/1593187612 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f1ff41041a0 con 0x7f1fe00391b0 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.710+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 msgr2=0x7f1fe003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.710+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 0x7f1fe003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0098c0 tx=0x7f1fdc006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.710+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 msgr2=0x7f1ff419d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.710+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009940 tx=0x7f1fe4032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 shutdown_connections 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f1fe00391b0 0x7f1fe003b670 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 --2- 172.21.15.78:0/1593187612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4107790 0x7f1ff419d970 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 >> 172.21.15.78:0/1593187612 conn(0x7f1ff4102fa0 msgr2=0x7f1ff4103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 shutdown_connections 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:17.711+0000 7f1ff96ab640 1 -- 172.21.15.78:0/1593187612 wait complete. 2024-09-16T01:01:18.238 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2024-09-16T01:01:18.980 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-mon[30937]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:19.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Generating public/private rsa key pair. 2024-09-16T01:01:19.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Your identification has been saved in /tmp/tmpjlyizd8_/key 2024-09-16T01:01:19.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Your public key has been saved in /tmp/tmpjlyizd8_/key.pub 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: The key fingerprint is: 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: SHA256:Q4pGc6x81tcjZZ0EPqfPnVvs7dV4DlrXLVDVqEQL4Uk ceph-fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: The key's randomart image is: 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: +---[RSA 3072]----+ 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | Eoo...o| 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | . o +.+.o.| 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | o o . o.*.= | 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | o = + +.= | 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | = + S o = | 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | . o o . = +=| 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | B.@| 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | o B=| 2024-09-16T01:01:19.279 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: | . o=| 2024-09-16T01:01:19.280 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: +----[SHA256]-----+ 2024-09-16T01:01:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f77c4640 1 Processor -- start 2024-09-16T01:01:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f77c4640 1 -- start start 2024-09-16T01:01:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f77c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f77c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0108160 con 0x7fd5f0107790 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f5560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f5560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51406/0 (socket says 172.21.15.78:51406) 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.637+0000 7fd5f5560640 1 -- 172.21.15.78:0/1390509337 learned_addr learned my addr 172.21.15.78:0/1390509337 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.638+0000 7fd5f5560640 1 -- 172.21.15.78:0/1390509337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0108940 con 0x7fd5f0107790 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5f5560640 1 --2- 172.21.15.78:0/1390509337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009820 tx=0x7fd5e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e47c90efca32c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5e7fff640 1 -- 172.21.15.78:0/1390509337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0037070 con 0x7fd5f0107790 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5e7fff640 1 -- 172.21.15.78:0/1390509337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5e0032900 con 0x7fd5f0107790 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5e7fff640 1 -- 172.21.15.78:0/1390509337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0030850 con 0x7fd5f0107790 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5f77c4640 1 -- 172.21.15.78:0/1390509337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 msgr2=0x7fd5f0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.639+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/1390509337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009820 tx=0x7fd5e002a270 comp rx=0 tx=0).stop 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- 172.21.15.78:0/1390509337 shutdown_connections 2024-09-16T01:01:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/1390509337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f0107b90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- 172.21.15.78:0/1390509337 >> 172.21.15.78:0/1390509337 conn(0x7fd5f0102fa0 msgr2=0x7fd5f01053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- 172.21.15.78:0/1390509337 shutdown_connections 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- 172.21.15.78:0/1390509337 wait complete. 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 Processor -- start 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- start start 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.640+0000 7fd5f77c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f010bcf0 con 0x7fd5f0107790 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f5560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f5560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51408/0 (socket says 172.21.15.78:51408) 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f5560640 1 -- 172.21.15.78:0/2239898077 learned_addr learned my addr 172.21.15.78:0/2239898077 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f5560640 1 -- 172.21.15.78:0/2239898077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f019bc80 con 0x7fd5f0107790 2024-09-16T01:01:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f5560640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00097f0 tx=0x7fd5e0030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0037040 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5e0040070 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f019bf10 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e003a390 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.641+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f019ec00 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.642+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fd5e003a630 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.642+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f0107c10 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.642+0000 7fd5e67fc640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 0x7fd5dc03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.642+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd5e006c7f0 con 0x7fd5f0107790 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.642+0000 7fd5f4d5f640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 0x7fd5dc03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.643+0000 7fd5f4d5f640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 0x7fd5dc03b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0098c0 tx=0x7fd5ec006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.647+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd5e004a200 con 0x7fd5f0107790 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.813+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd5f0104220 con 0x7fd5dc0391b0 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.985+0000 7fd5e67fc640 1 -- 172.21.15.78:0/2239898077 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd5f0104220 con 0x7fd5dc0391b0 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 msgr2=0x7fd5dc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 0x7fd5dc03b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0098c0 tx=0x7fd5ec006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 msgr2=0x7fd5f019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00097f0 tx=0x7fd5e0030d80 comp rx=0 tx=0).stop 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 shutdown_connections 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fd5dc0391b0 0x7fd5dc03b670 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 --2- 172.21.15.78:0/2239898077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f0107790 0x7fd5f019b740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 >> 172.21.15.78:0/2239898077 conn(0x7fd5f0102fa0 msgr2=0x7fd5f0103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 shutdown_connections 2024-09-16T01:01:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:18.987+0000 7fd5f77c4640 1 -- 172.21.15.78:0/2239898077 wait complete. 2024-09-16T01:01:20.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:19 smithi078 ceph-mon[30937]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:20.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:19 smithi078 ceph-mon[30937]: Generating ssh key... 2024-09-16T01:01:20.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:19 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:20.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:19 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:20.763 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.027+0000 7f439f7a3640 1 Processor -- start 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439f7a3640 1 -- start start 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439f7a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439f7a3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f439807d0a0 con 0x7f439807e6b0 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439d53f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439d53f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51422/0 (socket says 172.21.15.78:51422) 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439d53f640 1 -- 172.21.15.78:0/868935224 learned_addr learned my addr 172.21.15.78:0/868935224 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.028+0000 7f439d53f640 1 -- 172.21.15.78:0/868935224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439807d220 con 0x7f439807e6b0 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.029+0000 7f439d53f640 1 --2- 172.21.15.78:0/868935224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4394009820 tx=0x7f439402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67d83aafaac5f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.029+0000 7f438ffff640 1 -- 172.21.15.78:0/868935224 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4394037070 con 0x7f439807e6b0 2024-09-16T01:01:20.764 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.029+0000 7f438ffff640 1 -- 172.21.15.78:0/868935224 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4394032900 con 0x7f439807e6b0 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.029+0000 7f438ffff640 1 -- 172.21.15.78:0/868935224 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43940308e0 con 0x7f439807e6b0 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- 172.21.15.78:0/868935224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 msgr2=0x7f439807cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 --2- 172.21.15.78:0/868935224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4394009820 tx=0x7f439402a300 comp rx=0 tx=0).stop 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- 172.21.15.78:0/868935224 shutdown_connections 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 --2- 172.21.15.78:0/868935224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439807cb60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- 172.21.15.78:0/868935224 >> 172.21.15.78:0/868935224 conn(0x7f4398102fd0 msgr2=0x7f43981053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- 172.21.15.78:0/868935224 shutdown_connections 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- 172.21.15.78:0/868935224 wait complete. 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 Processor -- start 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- start start 2024-09-16T01:01:20.765 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.030+0000 7f439f7a3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f439810ddb0 con 0x7f439807e6b0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439d53f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439d53f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51432/0 (socket says 172.21.15.78:51432) 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439d53f640 1 -- 172.21.15.78:0/2403263465 learned_addr learned my addr 172.21.15.78:0/2403263465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439d53f640 1 -- 172.21.15.78:0/2403263465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439819de90 con 0x7f439807e6b0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439d53f640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4394030610 tx=0x7f4394032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4394037040 con 0x7f439807e6b0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4394040070 con 0x7f439807e6b0 2024-09-16T01:01:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f439403a390 con 0x7f439807e6b0 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439819e120 con 0x7f439807e6b0 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.031+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43981a0e10 con 0x7f439807e6b0 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.032+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f439403b650 con 0x7f439807e6b0 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.032+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439807cbe0 con 0x7f439807e6b0 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.032+0000 7f438e7fc640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 0x7f438403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:20.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.032+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f439406c4d0 con 0x7f439807e6b0 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.032+0000 7f439cd3e640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 0x7f438403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.033+0000 7f439cd3e640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 0x7f438403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43880098c0 tx=0x7f4388006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.037+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4394032900 con 0x7f439807e6b0 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.205+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f4398104280 con 0x7f4384039200 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.206+0000 7f438e7fc640 1 -- 172.21.15.78:0/2403263465 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f4398104280 con 0x7f4384039200 2024-09-16T01:01:20.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 msgr2=0x7f438403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 0x7f438403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43880098c0 tx=0x7f4388006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 msgr2=0x7f439819d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4394030610 tx=0x7f4394032670 comp rx=0 tx=0).stop 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 shutdown_connections 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f4384039200 0x7f438403b6c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 --2- 172.21.15.78:0/2403263465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f439807e6b0 0x7f439819d950 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:20.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 >> 172.21.15.78:0/2403263465 conn(0x7f4398102fd0 msgr2=0x7f4398103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 shutdown_connections 2024-09-16T01:01:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:20.208+0000 7f439f7a3640 1 -- 172.21.15.78:0/2403263465 wait complete. 2024-09-16T01:01:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-16T01:01:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2024-09-16T01:01:20.770 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2024-09-16T01:01:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:21 smithi078 ceph-mon[30937]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:22.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:22 smithi078 ceph-mon[30937]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:23 smithi078 ceph-mon[30937]: Deploying cephadm binary to smithi078 2024-09-16T01:01:25.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2024-09-16T01:01:25.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f170689d640 1 Processor -- start 2024-09-16T01:01:25.071 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f170689d640 1 -- start start 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f170689d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f170689d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1700108160 con 0x7f1700107790 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f16fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f16fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51448/0 (socket says 172.21.15.78:51448) 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f16fffff640 1 -- 172.21.15.78:0/537953385 learned_addr learned my addr 172.21.15.78:0/537953385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.203+0000 7f16fffff640 1 -- 172.21.15.78:0/537953385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1700108940 con 0x7f1700107790 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.204+0000 7f16fffff640 1 --2- 172.21.15.78:0/537953385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009820 tx=0x7f16f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77742b6df9cf9a92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.204+0000 7f16feffd640 1 -- 172.21.15.78:0/537953385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f0037070 con 0x7f1700107790 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.204+0000 7f16feffd640 1 -- 172.21.15.78:0/537953385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f16f0032900 con 0x7f1700107790 2024-09-16T01:01:25.072 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.204+0000 7f16feffd640 1 -- 172.21.15.78:0/537953385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f0030850 con 0x7f1700107790 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- 172.21.15.78:0/537953385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 msgr2=0x7f1700107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 --2- 172.21.15.78:0/537953385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009820 tx=0x7f16f002a270 comp rx=0 tx=0).stop 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- 172.21.15.78:0/537953385 shutdown_connections 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 --2- 172.21.15.78:0/537953385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f1700107b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- 172.21.15.78:0/537953385 >> 172.21.15.78:0/537953385 conn(0x7f1700102fa0 msgr2=0x7f17001053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- 172.21.15.78:0/537953385 shutdown_connections 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- 172.21.15.78:0/537953385 wait complete. 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 Processor -- start 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.205+0000 7f170689d640 1 -- start start 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f170689d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:25.073 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f170689d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f170010bcf0 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51450/0 (socket says 172.21.15.78:51450) 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fffff640 1 -- 172.21.15.78:0/521367230 learned_addr learned my addr 172.21.15.78:0/521367230 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fffff640 1 -- 172.21.15.78:0/521367230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170019bc80 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fffff640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009940 tx=0x7f16f0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f0042070 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f16f00308c0 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170019bf10 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.206+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16f0037040 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.207+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170019ec00 con 0x7f1700107790 2024-09-16T01:01:25.074 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.207+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f16f0045070 con 0x7f1700107790 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.207+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170010be70 con 0x7f1700107790 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.208+0000 7f16fd7fa640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 0x7f16e803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.208+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f16f006c320 con 0x7f1700107790 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.208+0000 7f16ff7fe640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 0x7f16e803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.208+0000 7f16ff7fe640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 0x7f16e803b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16f40098c0 tx=0x7f16f4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.212+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f16f004b170 con 0x7f1700107790 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:21.382+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) -- 0x7f1700104220 con 0x7f16e8039200 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.524+0000 7f16fd7fa640 1 -- 172.21.15.78:0/521367230 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1700104220 con 0x7f16e8039200 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 msgr2=0x7f16e803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 0x7f16e803b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16f40098c0 tx=0x7f16f4006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 msgr2=0x7f170019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009940 tx=0x7f16f0032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 shutdown_connections 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f16e8039200 0x7f16e803b6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 --2- 172.21.15.78:0/521367230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1700107790 0x7f170019b740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 >> 172.21.15.78:0/521367230 conn(0x7f1700102fa0 msgr2=0x7f1700103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 shutdown_connections 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:24.527+0000 7f170689d640 1 -- 172.21.15.78:0/521367230 wait complete. 2024-09-16T01:01:25.076 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2024-09-16T01:01:25.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:25 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:25.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:25 smithi078 ceph-mon[30937]: Added host smithi078 2024-09-16T01:01:25.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:25 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:26.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-16T01:01:26.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.531+0000 7f3302606640 1 Processor -- start 2024-09-16T01:01:26.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.531+0000 7f3302606640 1 -- start start 2024-09-16T01:01:26.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.531+0000 7f3302606640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.531+0000 7f3302606640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc108160 con 0x7f32fc107790 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.531+0000 7f32fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.532+0000 7f32fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44830/0 (socket says 172.21.15.78:44830) 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.532+0000 7f32fbfff640 1 -- 172.21.15.78:0/1323517548 learned_addr learned my addr 172.21.15.78:0/1323517548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.532+0000 7f32fbfff640 1 -- 172.21.15.78:0/1323517548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc108940 con 0x7f32fc107790 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f32fbfff640 1 --2- 172.21.15.78:0/1323517548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009820 tx=0x7f32ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765ec68c37010bc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f32faffd640 1 -- 172.21.15.78:0/1323517548 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ec037070 con 0x7f32fc107790 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f32faffd640 1 -- 172.21.15.78:0/1323517548 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f32ec032900 con 0x7f32fc107790 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f32faffd640 1 -- 172.21.15.78:0/1323517548 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ec030850 con 0x7f32fc107790 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 -- 172.21.15.78:0/1323517548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 msgr2=0x7f32fc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 --2- 172.21.15.78:0/1323517548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009820 tx=0x7f32ec02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 -- 172.21.15.78:0/1323517548 shutdown_connections 2024-09-16T01:01:26.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 --2- 172.21.15.78:0/1323517548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc107b90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 -- 172.21.15.78:0/1323517548 >> 172.21.15.78:0/1323517548 conn(0x7f32fc102fa0 msgr2=0x7f32fc1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 -- 172.21.15.78:0/1323517548 shutdown_connections 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.533+0000 7f3302606640 1 -- 172.21.15.78:0/1323517548 wait complete. 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f3302606640 1 Processor -- start 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f3302606640 1 -- start start 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f3302606640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f3302606640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc10bcf0 con 0x7f32fc107790 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f32fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f32fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44832/0 (socket says 172.21.15.78:44832) 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.534+0000 7f32fbfff640 1 -- 172.21.15.78:0/1522234859 learned_addr learned my addr 172.21.15.78:0/1522234859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f32fbfff640 1 -- 172.21.15.78:0/1522234859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc19bc80 con 0x7f32fc107790 2024-09-16T01:01:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f32fbfff640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009940 tx=0x7f32ec032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ec042070 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f32ec0308c0 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32fc19bf10 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32ec037040 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.535+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc19ec00 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f32ec045070 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f32f97fa640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 0x7f32e403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f32ec06c3a0 con 0x7f32fc107790 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f32fb7fe640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 0x7f32e403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:26.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32fc10be70 con 0x7f32fc107790 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.536+0000 7f32fb7fe640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 0x7f32e403b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32f00098c0 tx=0x7f32f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.541+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f32ec03cd90 con 0x7f32fc107790 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.722+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f32fc104220 con 0x7f32e40391b0 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.727+0000 7f32f97fa640 1 -- 172.21.15.78:0/1522234859 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f32fc104220 con 0x7f32e40391b0 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 msgr2=0x7f32e403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 0x7f32e403b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32f00098c0 tx=0x7f32f0006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 msgr2=0x7f32fc19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009940 tx=0x7f32ec032ce0 comp rx=0 tx=0).stop 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 shutdown_connections 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f32e40391b0 0x7f32e403b670 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 --2- 172.21.15.78:0/1522234859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32fc107790 0x7f32fc19b740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 >> 172.21.15.78:0/1522234859 conn(0x7f32fc102fa0 msgr2=0x7f32fc103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:26.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 shutdown_connections 2024-09-16T01:01:26.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:25.729+0000 7f3302606640 1 -- 172.21.15.78:0/1522234859 wait complete. 2024-09-16T01:01:26.294 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2024-09-16T01:01:26.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:26 smithi078 ceph-mon[30937]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:26.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:26 smithi078 ceph-mon[30937]: Saving service mon spec with placement count:5 2024-09-16T01:01:26.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:26 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:28 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:28 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:28 smithi078 ceph-mon[30937]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:28.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:28 smithi078 ceph-mon[30937]: Saving service mgr spec with placement count:2 2024-09-16T01:01:28.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:28 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:28.739 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.687+0000 7fead3bcd640 1 Processor -- start 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead3bcd640 1 -- start start 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead3bcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead3bcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feacc07c5a0 con 0x7feacc106060 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead1969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead1969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44834/0 (socket says 172.21.15.78:44834) 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.688+0000 7fead1969640 1 -- 172.21.15.78:0/45018208 learned_addr learned my addr 172.21.15.78:0/45018208 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.689+0000 7fead1969640 1 -- 172.21.15.78:0/45018208 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc1069a0 con 0x7feacc106060 2024-09-16T01:01:28.740 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.689+0000 7fead1969640 1 --2- 172.21.15.78:0/45018208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feac8009820 tx=0x7feac802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f1df82154ac7b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.689+0000 7fead0967640 1 -- 172.21.15.78:0/45018208 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac8037070 con 0x7feacc106060 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.689+0000 7fead0967640 1 -- 172.21.15.78:0/45018208 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac8032920 con 0x7feacc106060 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.689+0000 7fead0967640 1 -- 172.21.15.78:0/45018208 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac8030810 con 0x7feacc106060 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.690+0000 7fead3bcd640 1 -- 172.21.15.78:0/45018208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 msgr2=0x7feacc106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.690+0000 7fead3bcd640 1 --2- 172.21.15.78:0/45018208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feac8009820 tx=0x7feac802a270 comp rx=0 tx=0).stop 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.690+0000 7fead3bcd640 1 -- 172.21.15.78:0/45018208 shutdown_connections 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.690+0000 7fead3bcd640 1 --2- 172.21.15.78:0/45018208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc106460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.690+0000 7fead3bcd640 1 -- 172.21.15.78:0/45018208 >> 172.21.15.78:0/45018208 conn(0x7feacc101e70 msgr2=0x7feacc104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.691+0000 7fead3bcd640 1 -- 172.21.15.78:0/45018208 shutdown_connections 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.691+0000 7fead3bcd640 1 -- 172.21.15.78:0/45018208 wait complete. 2024-09-16T01:01:28.741 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.691+0000 7fead3bcd640 1 Processor -- start 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.691+0000 7fead3bcd640 1 -- start start 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead3bcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead3bcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feacc107ae0 con 0x7feacc106060 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead1969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead1969640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44840/0 (socket says 172.21.15.78:44840) 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead1969640 1 -- 172.21.15.78:0/105811640 learned_addr learned my addr 172.21.15.78:0/105811640 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead1969640 1 -- 172.21.15.78:0/105811640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc19de70 con 0x7feacc106060 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead1969640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feac80097f0 tx=0x7feac80327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac8037040 con 0x7feacc106060 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac8030cc0 con 0x7feacc106060 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feacc19e100 con 0x7feacc106060 2024-09-16T01:01:28.742 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.692+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feacc1a0df0 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.693+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac803a6b0 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.693+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7feac803a8c0 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.693+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feacc106460 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.694+0000 7feac2ffd640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 0x7feab803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.694+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feac806ca60 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.694+0000 7fead1168640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 0x7feab803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.694+0000 7fead1168640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 0x7feab803b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feabc0098c0 tx=0x7feabc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.698+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feac80371e0 con 0x7feacc106060 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.902+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7feacc02da20 con 0x7feab80391b0 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.910+0000 7feac2ffd640 1 -- 172.21.15.78:0/105811640 <== mgr.14118 v2:172.21.15.78:6800/2425087015 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7feacc02da20 con 0x7feab80391b0 2024-09-16T01:01:28.743 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 msgr2=0x7feab803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 0x7feab803b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7feabc0098c0 tx=0x7feabc006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 msgr2=0x7feacc19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feac80097f0 tx=0x7feac80327b0 comp rx=0 tx=0).stop 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 shutdown_connections 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feab80391b0 0x7feab803b670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 --2- 172.21.15.78:0/105811640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feacc106060 0x7feacc19d930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.912+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 >> 172.21.15.78:0/105811640 conn(0x7feacc101e70 msgr2=0x7feacc1945b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.913+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 shutdown_connections 2024-09-16T01:01:28.744 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:27.913+0000 7fead3bcd640 1 -- 172.21.15.78:0/105811640 wait complete. 2024-09-16T01:01:30.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.512+0000 7feee5c52640 1 Processor -- start 2024-09-16T01:01:30.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.512+0000 7feee5c52640 1 -- start start 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.512+0000 7feee5c52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.513+0000 7feee5c52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feee0107010 con 0x7feee0106640 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.513+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.513+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44856/0 (socket says 172.21.15.78:44856) 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.513+0000 7feedf7fe640 1 -- 172.21.15.78:0/1004644511 learned_addr learned my addr 172.21.15.78:0/1004644511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.513+0000 7feedf7fe640 1 -- 172.21.15.78:0/1004644511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee01077f0 con 0x7feee0106640 2024-09-16T01:01:30.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.514+0000 7feedf7fe640 1 --2- 172.21.15.78:0/1004644511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feed0009820 tx=0x7feed002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515610dd8d83928e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.514+0000 7feede7fc640 1 -- 172.21.15.78:0/1004644511 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feed0037070 con 0x7feee0106640 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.514+0000 7feede7fc640 1 -- 172.21.15.78:0/1004644511 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed0032920 con 0x7feee0106640 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.514+0000 7feede7fc640 1 -- 172.21.15.78:0/1004644511 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feed0030810 con 0x7feee0106640 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 -- 172.21.15.78:0/1004644511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 msgr2=0x7feee0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 --2- 172.21.15.78:0/1004644511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feed0009820 tx=0x7feed002a270 comp rx=0 tx=0).stop 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 -- 172.21.15.78:0/1004644511 shutdown_connections 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 --2- 172.21.15.78:0/1004644511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee0106a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 -- 172.21.15.78:0/1004644511 >> 172.21.15.78:0/1004644511 conn(0x7feee0101db0 msgr2=0x7feee0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 -- 172.21.15.78:0/1004644511 shutdown_connections 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 -- 172.21.15.78:0/1004644511 wait complete. 2024-09-16T01:01:30.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.515+0000 7feee5c52640 1 Processor -- start 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feee5c52640 1 -- start start 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feee5c52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feee5c52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feee010aba0 con 0x7feee0106640 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44860/0 (socket says 172.21.15.78:44860) 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedf7fe640 1 -- 172.21.15.78:0/196595193 learned_addr learned my addr 172.21.15.78:0/196595193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedf7fe640 1 -- 172.21.15.78:0/196595193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee019df20 con 0x7feee0106640 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedf7fe640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feed00097f0 tx=0x7feed00327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feed0037040 con 0x7feee0106640 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.516+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee019e1b0 con 0x7feee0106640 2024-09-16T01:01:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.517+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feed0030c40 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.517+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feed003a5b0 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.517+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee01a0ea0 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.517+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7feed003a7a0 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.518+0000 7feedcff9640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 0x7feec803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.518+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feed006c720 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.518+0000 7feedeffd640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 0x7feec803b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.518+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee010ad20 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.518+0000 7feedeffd640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 0x7feec803b720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feed40098c0 tx=0x7feed4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.523+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feed0030df0 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.696+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7feee002d900 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.698+0000 7feedcff9640 1 -- 172.21.15.78:0/196595193 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7feed006c070 con 0x7feee0106640 2024-09-16T01:01:30.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 msgr2=0x7feec803b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 0x7feec803b720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feed40098c0 tx=0x7feed4006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 msgr2=0x7feee019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feed00097f0 tx=0x7feed00327b0 comp rx=0 tx=0).stop 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 shutdown_connections 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7feec8039260 0x7feec803b720 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 --2- 172.21.15.78:0/196595193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feee0106640 0x7feee019d9e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 >> 172.21.15.78:0/196595193 conn(0x7feee0101db0 msgr2=0x7feee0102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 shutdown_connections 2024-09-16T01:01:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:29.701+0000 7feee5c52640 1 -- 172.21.15.78:0/196595193 wait complete. 2024-09-16T01:01:30.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:30 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:30.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:30 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/196595193' entity='client.admin' 2024-09-16T01:01:30.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:30 smithi078 ceph-mon[30937]: from='mgr.14118 172.21.15.78:0/1685268120' entity='mgr.x' 2024-09-16T01:01:32.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc54a49d640 1 Processor -- start 2024-09-16T01:01:32.107 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc54a49d640 1 -- start start 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc54a49d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc54a49d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc544108160 con 0x7fc544107790 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44870/0 (socket says 172.21.15.78:44870) 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.772+0000 7fc543fff640 1 -- 172.21.15.78:0/3227864975 learned_addr learned my addr 172.21.15.78:0/3227864975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.773+0000 7fc543fff640 1 -- 172.21.15.78:0/3227864975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc544108940 con 0x7fc544107790 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.773+0000 7fc543fff640 1 --2- 172.21.15.78:0/3227864975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc534009820 tx=0x7fc53402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4326923a262f8e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.773+0000 7fc5437fe640 1 -- 172.21.15.78:0/3227864975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc534037070 con 0x7fc544107790 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.773+0000 7fc5437fe640 1 -- 172.21.15.78:0/3227864975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc534032920 con 0x7fc544107790 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.773+0000 7fc5437fe640 1 -- 172.21.15.78:0/3227864975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc534030810 con 0x7fc544107790 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 -- 172.21.15.78:0/3227864975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 msgr2=0x7fc544107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 --2- 172.21.15.78:0/3227864975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc534009820 tx=0x7fc53402a270 comp rx=0 tx=0).stop 2024-09-16T01:01:32.108 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 -- 172.21.15.78:0/3227864975 shutdown_connections 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 --2- 172.21.15.78:0/3227864975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc544107b90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 -- 172.21.15.78:0/3227864975 >> 172.21.15.78:0/3227864975 conn(0x7fc544102fa0 msgr2=0x7fc5441053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 -- 172.21.15.78:0/3227864975 shutdown_connections 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.774+0000 7fc54a49d640 1 -- 172.21.15.78:0/3227864975 wait complete. 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc54a49d640 1 Processor -- start 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc54a49d640 1 -- start start 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc54a49d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc54a49d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410bcf0 con 0x7fc544107790 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44874/0 (socket says 172.21.15.78:44874) 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc543fff640 1 -- 172.21.15.78:0/1843536034 learned_addr learned my addr 172.21.15.78:0/1843536034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc543fff640 1 -- 172.21.15.78:0/1843536034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54419bca0 con 0x7fc544107790 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc543fff640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc534030560 tx=0x7fc534032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:32.109 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.775+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc534037040 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.776+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc534040070 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.776+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc54419bf30 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.776+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc53403a740 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.776+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc54419ec20 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.776+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fc53403b420 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.777+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc54410be70 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.777+0000 7fc541ffb640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 0x7fc53003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.777+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc53406c820 con 0x7fc544107790 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.777+0000 7fc53bfff640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 0x7fc53003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.777+0000 7fc53bfff640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 0x7fc53003b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0098c0 tx=0x7fc52c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:32.110 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.781+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc53404a200 con 0x7fc544107790 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.980+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fc544107c10 con 0x7fc544107790 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.982+0000 7fc541ffb640 1 -- 172.21.15.78:0/1843536034 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fc534031e80 con 0x7fc544107790 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 msgr2=0x7fc53003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 0x7fc53003b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0098c0 tx=0x7fc52c006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 msgr2=0x7fc54419b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc534030560 tx=0x7fc534032670 comp rx=0 tx=0).stop 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 shutdown_connections 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc530039200 0x7fc53003b6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 --2- 172.21.15.78:0/1843536034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc544107790 0x7fc54419b760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 >> 172.21.15.78:0/1843536034 conn(0x7fc544102fa0 msgr2=0x7fc544103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 shutdown_connections 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:30.984+0000 7fc54a49d640 1 -- 172.21.15.78:0/1843536034 wait complete. 2024-09-16T01:01:32.111 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2024-09-16T01:01:32.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:31 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1843536034' entity='client.admin' 2024-09-16T01:01:33.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:32 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/687484289' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-16T01:01:33.236 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setuser ceph since I am not root 2024-09-16T01:01:33.236 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setgroup ceph since I am not root 2024-09-16T01:01:33.236 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.132+0000 7f3f6ffe4040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:01:33.236 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.188+0000 7f3f6ffe4040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:01:33.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.338+0000 7f3f6ffe4040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:01:33.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.405+0000 7f3f6ffe4040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:01:33.638 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777c8be640 1 Processor -- start 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777c8be640 1 -- start start 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777c8be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777c8be640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f777807d060 con 0x7f777807e670 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44880/0 (socket says 172.21.15.78:44880) 2024-09-16T01:01:33.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777659c640 1 -- 172.21.15.78:0/1638272120 learned_addr learned my addr 172.21.15.78:0/1638272120 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.498+0000 7f777659c640 1 -- 172.21.15.78:0/1638272120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777807d1e0 con 0x7f777807e670 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.499+0000 7f777659c640 1 --2- 172.21.15.78:0/1638272120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7768009820 tx=0x7f776802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb22329017c55c6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.499+0000 7f7775d9b640 1 -- 172.21.15.78:0/1638272120 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7768037070 con 0x7f777807e670 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.499+0000 7f7775d9b640 1 -- 172.21.15.78:0/1638272120 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768032920 con 0x7f777807e670 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.499+0000 7f7775d9b640 1 -- 172.21.15.78:0/1638272120 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7768030810 con 0x7f777807e670 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 -- 172.21.15.78:0/1638272120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 msgr2=0x7f777807cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 --2- 172.21.15.78:0/1638272120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7768009820 tx=0x7f776802a270 comp rx=0 tx=0).stop 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 -- 172.21.15.78:0/1638272120 shutdown_connections 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 --2- 172.21.15.78:0/1638272120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777807cb20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 -- 172.21.15.78:0/1638272120 >> 172.21.15.78:0/1638272120 conn(0x7f7778102fa0 msgr2=0x7f77781053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 -- 172.21.15.78:0/1638272120 shutdown_connections 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.500+0000 7f777c8be640 1 -- 172.21.15.78:0/1638272120 wait complete. 2024-09-16T01:01:33.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777c8be640 1 Processor -- start 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777c8be640 1 -- start start 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777c8be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777c8be640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f777810dde0 con 0x7f777807e670 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44886/0 (socket says 172.21.15.78:44886) 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777659c640 1 -- 172.21.15.78:0/687484289 learned_addr learned my addr 172.21.15.78:0/687484289 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777659c640 1 -- 172.21.15.78:0/687484289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777819de70 con 0x7f777807e670 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.501+0000 7f777659c640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7768030380 tx=0x7f7768032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7768037040 con 0x7f777807e670 2024-09-16T01:01:33.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777819e100 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768040070 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f776803a7b0 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77781a0df0 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.502+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f776803aa30 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.503+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777807cba0 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.503+0000 7f776f7fe640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 0x7f776403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.503+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f776806ca80 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.503+0000 7f776ffff640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 0x7f776403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.503+0000 7f776ffff640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 0x7f776403b670 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77600098c0 tx=0x7f7760006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.507+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7768044c70 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.729+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f77781a0fc0 con 0x7f777807e670 2024-09-16T01:01:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.986+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f7768042d80 con 0x7f777807e670 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.987+0000 7f776f7fe640 1 -- 172.21.15.78:0/687484289 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f7768045460 con 0x7f777807e670 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 msgr2=0x7f776403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 0x7f776403b670 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f77600098c0 tx=0x7f7760006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 msgr2=0x7f777819d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7768030380 tx=0x7f7768032670 comp rx=0 tx=0).stop 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 shutdown_connections 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f77640391b0 0x7f776403b670 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 --2- 172.21.15.78:0/687484289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f777807e670 0x7f777819d930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 >> 172.21.15.78:0/687484289 conn(0x7f7778102fa0 msgr2=0x7f7778103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 shutdown_connections 2024-09-16T01:01:33.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:32.989+0000 7f777c8be640 1 -- 172.21.15.78:0/687484289 wait complete. 2024-09-16T01:01:33.868 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.553+0000 7f3f6ffe4040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:01:33.868 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:33.706+0000 7f3f6ffe4040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:01:34.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/687484289' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-16T01:01:34.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:33 smithi078 ceph-mon[30937]: mgrmap e9: x(active, since 20s) 2024-09-16T01:01:34.255 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:34.031+0000 7f3f6ffe4040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:01:34.255 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:34.096+0000 7f3f6ffe4040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:01:34.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:34.290+0000 7f3f6ffe4040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:01:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f7162a2f640 1 Processor -- start 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f7162a2f640 1 -- start start 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f7162a2f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f7162a2f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f715c07bdf0 con 0x7f715c1081f0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f715bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f715bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44918/0 (socket says 172.21.15.78:44918) 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.047+0000 7f715bfff640 1 -- 172.21.15.78:0/2439160849 learned_addr learned my addr 172.21.15.78:0/2439160849 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.048+0000 7f715bfff640 1 -- 172.21.15.78:0/2439160849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c10ab20 con 0x7f715c1081f0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.048+0000 7f715bfff640 1 --2- 172.21.15.78:0/2439160849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f714c009820 tx=0x7f714c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d018f8f4eeeb9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.048+0000 7f715affd640 1 -- 172.21.15.78:0/2439160849 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f714c037070 con 0x7f715c1081f0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.048+0000 7f715affd640 1 -- 172.21.15.78:0/2439160849 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f714c032920 con 0x7f715c1081f0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.048+0000 7f715affd640 1 -- 172.21.15.78:0/2439160849 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f714c030810 con 0x7f715c1081f0 2024-09-16T01:01:34.951 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 -- 172.21.15.78:0/2439160849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 msgr2=0x7f715c10a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 --2- 172.21.15.78:0/2439160849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f714c009820 tx=0x7f714c02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 -- 172.21.15.78:0/2439160849 shutdown_connections 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 --2- 172.21.15.78:0/2439160849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c10a5e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 -- 172.21.15.78:0/2439160849 >> 172.21.15.78:0/2439160849 conn(0x7f715c101db0 msgr2=0x7f715c104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.049+0000 7f7162a2f640 1 -- 172.21.15.78:0/2439160849 shutdown_connections 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.050+0000 7f7162a2f640 1 -- 172.21.15.78:0/2439160849 wait complete. 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.050+0000 7f7162a2f640 1 Processor -- start 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.050+0000 7f7162a2f640 1 -- start start 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.050+0000 7f7162a2f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.050+0000 7f7162a2f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f715c10bb40 con 0x7f715c1081f0 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f715bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f715bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44934/0 (socket says 172.21.15.78:44934) 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f715bfff640 1 -- 172.21.15.78:0/4090781417 learned_addr learned my addr 172.21.15.78:0/4090781417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f715bfff640 1 -- 172.21.15.78:0/4090781417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c19bcd0 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f715bfff640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f714c030420 tx=0x7f714c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f714c037040 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f714c040070 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f715c19bf60 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f714c03a6e0 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.051+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f715c19ec50 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.052+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f714c03a9e0 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.052+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f715c108650 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.053+0000 7f71597fa640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 0x7f714403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.053+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f714c06cc60 con 0x7f715c1081f0 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.053+0000 7f715b7fe640 1 -- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 msgr2=0x7f714403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:34.953 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.053+0000 7f715b7fe640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 0x7f714403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.057+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f714c03ae20 con 0x7f715c1081f0 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.254+0000 7f715b7fe640 1 -- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 msgr2=0x7f714403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.254+0000 7f715b7fe640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 0x7f714403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.279+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f715c19ee20 con 0x7f715c1081f0 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.280+0000 7f71597fa640 1 -- 172.21.15.78:0/4090781417 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f715c19ee20 con 0x7f715c1081f0 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 msgr2=0x7f714403b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 0x7f714403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 msgr2=0x7f715c19b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f714c030420 tx=0x7f714c032670 comp rx=0 tx=0).stop 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 shutdown_connections 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7f7144039200 0x7f714403b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 --2- 172.21.15.78:0/4090781417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c1081f0 0x7f715c19b790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:34.954 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 >> 172.21.15.78:0/4090781417 conn(0x7f715c101db0 msgr2=0x7f715c102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:34.955 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 shutdown_connections 2024-09-16T01:01:34.955 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:34.282+0000 7f7162a2f640 1 -- 172.21.15.78:0/4090781417 wait complete. 2024-09-16T01:01:34.955 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-09-16T01:01:34.955 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2024-09-16T01:01:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:34 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4090781417' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-16T01:01:35.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:35.204+0000 7f3f6ffe4040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:01:35.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:35.277+0000 7f3f6ffe4040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:01:36.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.021+0000 7f3f6ffe4040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:01:36.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.140+0000 7f3f6ffe4040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:01:36.575 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.287+0000 7f3f6ffe4040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:01:36.575 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.350+0000 7f3f6ffe4040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:01:36.576 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.505+0000 7f3f6ffe4040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:01:36.576 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.574+0000 7f3f6ffe4040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:01:36.859 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.634+0000 7f3f6ffe4040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:01:36.859 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.793+0000 7f3f6ffe4040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:01:36.859 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.858+0000 7f3f6ffe4040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:01:37.125 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:36.923+0000 7f3f6ffe4040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:01:37.125 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:37.005+0000 7f3f6ffe4040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:01:37.125 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:37.065+0000 7f3f6ffe4040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:01:37.125 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:37.124+0000 7f3f6ffe4040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:01:38.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:37.910+0000 7f3f6ffe4040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:01:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: Active manager daemon x restarted 2024-09-16T01:01:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: Activating manager daemon x 2024-09-16T01:01:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: osdmap e3: 0 total, 0 up, 0 in 2024-09-16T01:01:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: mgrmap e10: x(active, starting, since 0.00348957s) 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: Manager daemon x is now available 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:01:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:01:39.518 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:39.294+0000 7f3ed5aad640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:01:01:39] ENGINE Error in HTTPServer.serve 2024-09-16T01:01:39.518 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Traceback (most recent call last): 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._connections.run(self.expiration_interval) 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._run(expiration_interval) 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s = self.context.wrap_socket( 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: return self.sslsocket_class._create( 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:01:39.519 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self.do_handshake() 2024-09-16T01:01:39.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:01:39.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._sslobj.do_handshake() 2024-09-16T01:01:39.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:01:39.520 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:01:39.793 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-09-16T01:01:39.793 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b2ad1640 1 Processor -- start 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b2ad1640 1 -- start start 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b2ad1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b2ad1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac108130 con 0x7fc6ac107760 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b086d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b086d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46050/0 (socket says 172.21.15.78:46050) 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b086d640 1 -- 172.21.15.78:0/805504377 learned_addr learned my addr 172.21.15.78:0/805504377 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.399+0000 7fc6b086d640 1 -- 172.21.15.78:0/805504377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac108910 con 0x7fc6ac107760 2024-09-16T01:01:39.794 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.400+0000 7fc6b086d640 1 --2- 172.21.15.78:0/805504377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009820 tx=0x7fc6a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7739f22d1daf0c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.400+0000 7fc6ab7fe640 1 -- 172.21.15.78:0/805504377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a0037070 con 0x7fc6ac107760 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.400+0000 7fc6ab7fe640 1 -- 172.21.15.78:0/805504377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0032920 con 0x7fc6ac107760 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.400+0000 7fc6ab7fe640 1 -- 172.21.15.78:0/805504377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a0030810 con 0x7fc6ac107760 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/805504377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 msgr2=0x7fc6ac107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/805504377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009820 tx=0x7fc6a002a270 comp rx=0 tx=0).stop 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/805504377 shutdown_connections 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/805504377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac107b60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/805504377 >> 172.21.15.78:0/805504377 conn(0x7fc6ac102f30 msgr2=0x7fc6ac105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/805504377 shutdown_connections 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.401+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/805504377 wait complete. 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b2ad1640 1 Processor -- start 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b2ad1640 1 -- start start 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b2ad1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:39.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b2ad1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10bcc0 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b086d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b086d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46054/0 (socket says 172.21.15.78:46054) 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.402+0000 7fc6b086d640 1 -- 172.21.15.78:0/3199160779 learned_addr learned my addr 172.21.15.78:0/3199160779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6b086d640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac19df10 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6b086d640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00097f0 tx=0x7fc6a00327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a0037040 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac19e1a0 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0030c40 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a003a5b0 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.403+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac1a0e90 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.404+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fc6a003a7f0 con 0x7fc6ac107760 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.404+0000 7fc6a9ffb640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.404+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc69c03bdd0 con 0x7fc69c039200 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.404+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc6a006c740 con 0x7fc6ac107760 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.405+0000 7fc6abfff640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 msgr2=0x7fc69c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.405+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.606+0000 7fc6abfff640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 msgr2=0x7fc69c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:35.606+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:36.007+0000 7fc6abfff640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 msgr2=0x7fc69c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:36.007+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:36.808+0000 7fc6abfff640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 msgr2=0x7fc69c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2425087015 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:36.808+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:37.913+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7fc6a006b1f0 con 0x7fc6ac107760 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:37.913+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 msgr2=0x7fc69c03b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:37.913+0000 7fc6a9ffb640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.915+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fc6a006b4e0 con 0x7fc6ac107760 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.915+0000 7fc6a9ffb640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 0x7fc69c03f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.915+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc69c03bdd0 con 0x7fc69c03ccc0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.915+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 0x7fc69c03f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.916+0000 7fc6abfff640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 0x7fc69c03f0b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc698003980 tx=0x7fc698006e10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.916+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fc69c03bdd0 con 0x7fc69c03ccc0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.923+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc6ac10be40 con 0x7fc69c03ccc0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.923+0000 7fc6a9ffb640 1 -- 172.21.15.78:0/3199160779 <== mgr.14150 v2:172.21.15.78:6800/1698663383 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fc6ac10be40 con 0x7fc69c03ccc0 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 msgr2=0x7fc69c03f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 0x7fc69c03f0b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc698003980 tx=0x7fc698006e10 comp rx=0 tx=0).stop 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 msgr2=0x7fc6ac19d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00097f0 tx=0x7fc6a00327b0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 shutdown_connections 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc69c03ccc0 0x7fc69c03f0b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:6800/2425087015,v1:172.21.15.78:6801/2425087015] conn(0x7fc69c039200 0x7fc69c03b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 --2- 172.21.15.78:0/3199160779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6ac107760 0x7fc6ac19d9d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 >> 172.21.15.78:0/3199160779 conn(0x7fc6ac102f30 msgr2=0x7fc6ac103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 shutdown_connections 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:38.924+0000 7fc6b2ad1640 1 -- 172.21.15.78:0/3199160779 wait complete. 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2024-09-16T01:01:39.799 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2024-09-16T01:01:40.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-mon[30937]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-16T01:01:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-mon[30937]: mgrmap e11: x(active, since 1.00569s) 2024-09-16T01:01:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-mon[30937]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-16T01:01:40.040 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.217+0000 7f0f482a0640 1 Processor -- start 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.217+0000 7f0f482a0640 1 -- start start 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.217+0000 7f0f482a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.217+0000 7f0f482a0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4007bdf0 con 0x7f0f401081f0 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.218+0000 7f0f4603c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.218+0000 7f0f4603c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46140/0 (socket says 172.21.15.78:46140) 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.218+0000 7f0f4603c640 1 -- 172.21.15.78:0/3274953111 learned_addr learned my addr 172.21.15.78:0/3274953111 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.218+0000 7f0f4603c640 1 -- 172.21.15.78:0/3274953111 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4010ab20 con 0x7f0f401081f0 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f4603c640 1 --2- 172.21.15.78:0/3274953111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009820 tx=0x7f0f3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b3521a50d9b796 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f4503a640 1 -- 172.21.15.78:0/3274953111 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f3c037070 con 0x7f0f401081f0 2024-09-16T01:01:41.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f4503a640 1 -- 172.21.15.78:0/3274953111 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c032920 con 0x7f0f401081f0 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f4503a640 1 -- 172.21.15.78:0/3274953111 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f3c030810 con 0x7f0f401081f0 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f482a0640 1 -- 172.21.15.78:0/3274953111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 msgr2=0x7f0f4010a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.219+0000 7f0f482a0640 1 --2- 172.21.15.78:0/3274953111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c009820 tx=0x7f0f3c02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 -- 172.21.15.78:0/3274953111 shutdown_connections 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 --2- 172.21.15.78:0/3274953111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f4010a5e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 -- 172.21.15.78:0/3274953111 >> 172.21.15.78:0/3274953111 conn(0x7f0f40101db0 msgr2=0x7f0f40104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 -- 172.21.15.78:0/3274953111 shutdown_connections 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 -- 172.21.15.78:0/3274953111 wait complete. 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 Processor -- start 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.220+0000 7f0f482a0640 1 -- start start 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f482a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f482a0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4010bb40 con 0x7f0f401081f0 2024-09-16T01:01:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f4603c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f4603c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46142/0 (socket says 172.21.15.78:46142) 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f4603c640 1 -- 172.21.15.78:0/738660187 learned_addr learned my addr 172.21.15.78:0/738660187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f4603c640 1 -- 172.21.15.78:0/738660187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f40107fb0 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f4603c640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c030420 tx=0x7f0f3c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f3c037040 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c040070 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f3c03a6e0 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f401061a0 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.221+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f401066e0 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.222+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7f0f3c03a990 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.222+0000 7f0f377fe640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 0x7f0f2c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.222+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0f3c06c7f0 con 0x7f0f401081f0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.222+0000 7f0f4583b640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 0x7f0f2c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:41.093 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.222+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f40108650 con 0x7f0f401081f0 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.223+0000 7f0f4583b640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 0x7f0f2c03b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098c0 tx=0x7f0f30006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.227+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0f3c041270 con 0x7f0f401081f0 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.406+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f0f40106dd0 con 0x7f0f2c039140 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.410+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f0f3c045980 con 0x7f0f401081f0 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.493+0000 7f0f377fe640 1 -- 172.21.15.78:0/738660187 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f0f40106dd0 con 0x7f0f2c039140 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 msgr2=0x7f0f2c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 0x7f0f2c03b600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0f300098c0 tx=0x7f0f30006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 msgr2=0x7f0f40107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c030420 tx=0x7f0f3c032670 comp rx=0 tx=0).stop 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 shutdown_connections 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0f2c039140 0x7f0f2c03b600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 --2- 172.21.15.78:0/738660187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f401081f0 0x7f0f40107a70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:41.094 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.495+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 >> 172.21.15.78:0/738660187 conn(0x7f0f40101db0 msgr2=0x7f0f40102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:41.095 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.496+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 shutdown_connections 2024-09-16T01:01:41.095 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:40.496+0000 7f0f482a0640 1 -- 172.21.15.78:0/738660187 wait complete. 2024-09-16T01:01:41.095 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2024-09-16T01:01:41.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:39] ENGINE Bus STARTING 2024-09-16T01:01:41.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:39] ENGINE Serving on http://172.21.15.78:8765 2024-09-16T01:01:41.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:39] ENGINE Serving on https://172.21.15.78:7150 2024-09-16T01:01:41.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:39] ENGINE Bus STARTED 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: [16/Sep/2024:01:01:39] ENGINE Error in HTTPServer.serve 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-16T01:01:41.194 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: mgrmap e12: x(active, since 2s) 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:41.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:42.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:41 smithi078 ceph-mon[30937]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:42.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:41 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$tIfperOnjGzEVhVeqYw80.LQGrPaZML/dnmmdSfLB5ito0X30XqTS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726448501, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.482+0000 7fd5c5dc9640 1 Processor -- start 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.482+0000 7fd5c5dc9640 1 -- start start 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.482+0000 7fd5c5dc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.483+0000 7fd5c5dc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0108160 con 0x7fd5c0107790 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.483+0000 7fd5bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.483+0000 7fd5bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46156/0 (socket says 172.21.15.78:46156) 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.483+0000 7fd5bf7fe640 1 -- 172.21.15.78:0/2551229367 learned_addr learned my addr 172.21.15.78:0/2551229367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.483+0000 7fd5bf7fe640 1 -- 172.21.15.78:0/2551229367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c0108940 con 0x7fd5c0107790 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5bf7fe640 1 --2- 172.21.15.78:0/2551229367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009e80 tx=0x7fd5b002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88703a335e3b433c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5be7fc640 1 -- 172.21.15.78:0/2551229367 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5b0036070 con 0x7fd5c0107790 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5be7fc640 1 -- 172.21.15.78:0/2551229367 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b00318e0 con 0x7fd5c0107790 2024-09-16T01:01:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5be7fc640 1 -- 172.21.15.78:0/2551229367 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5b0007a00 con 0x7fd5c0107790 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2551229367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 msgr2=0x7fd5c0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2551229367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009e80 tx=0x7fd5b002a7a0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2551229367 shutdown_connections 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2551229367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c0107b90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2551229367 >> 172.21.15.78:0/2551229367 conn(0x7fd5c0102fa0 msgr2=0x7fd5c01053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2551229367 shutdown_connections 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.484+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2551229367 wait complete. 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.485+0000 7fd5c5dc9640 1 Processor -- start 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.485+0000 7fd5c5dc9640 1 -- start start 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.485+0000 7fd5c5dc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.485+0000 7fd5c5dc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c010bcf0 con 0x7fd5c0107790 2024-09-16T01:01:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.485+0000 7fd5bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46170/0 (socket says 172.21.15.78:46170) 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bf7fe640 1 -- 172.21.15.78:0/2034933 learned_addr learned my addr 172.21.15.78:0/2034933 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bf7fe640 1 -- 172.21.15.78:0/2034933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c019bca0 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bf7fe640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0031cc0 tx=0x7fd5b0031cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5b0036050 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b0042070 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c019bf30 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5b003d8e0 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.486+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c019ec20 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.487+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fd5b003dbe0 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.487+0000 7fd5bcff9640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 0x7fd5a803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.487+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd5b003bc80 con 0x7fd5c0107790 2024-09-16T01:01:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.487+0000 7fd5beffd640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 0x7fd5a803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.487+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c010be70 con 0x7fd5c0107790 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.488+0000 7fd5beffd640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 0x7fd5a803b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098c0 tx=0x7fd5b4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.492+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd5b0044310 con 0x7fd5c0107790 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.682+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd5c0107c10 con 0x7fd5a8039200 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.907+0000 7fd5bcff9640 1 -- 172.21.15.78:0/2034933 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd5c0107c10 con 0x7fd5a8039200 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 msgr2=0x7fd5a803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 0x7fd5a803b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40098c0 tx=0x7fd5b4006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 msgr2=0x7fd5c019b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0031cc0 tx=0x7fd5b0031cf0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 shutdown_connections 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd5a8039200 0x7fd5a803b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 --2- 172.21.15.78:0/2034933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5c0107790 0x7fd5c019b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:42.830 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 >> 172.21.15.78:0/2034933 conn(0x7fd5c0102fa0 msgr2=0x7fd5c0103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:42.831 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 shutdown_connections 2024-09-16T01:01:42.831 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:41.910+0000 7fd5c5dc9640 1 -- 172.21.15.78:0/2034933 wait complete. 2024-09-16T01:01:42.831 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2024-09-16T01:01:43.071 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:42 smithi078 ceph-mon[30937]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:44.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2024-09-16T01:01:44.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.242+0000 7fcd5e0c7640 1 Processor -- start 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.242+0000 7fcd5e0c7640 1 -- start start 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.242+0000 7fcd5e0c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.242+0000 7fcd5e0c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5807be40 con 0x7fcd581081b0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.243+0000 7fcd577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.243+0000 7fcd577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46184/0 (socket says 172.21.15.78:46184) 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.243+0000 7fcd577fe640 1 -- 172.21.15.78:0/487199268 learned_addr learned my addr 172.21.15.78:0/487199268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.243+0000 7fcd577fe640 1 -- 172.21.15.78:0/487199268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5810aae0 con 0x7fcd581081b0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd577fe640 1 --2- 172.21.15.78:0/487199268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009820 tx=0x7fcd4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f09352dcee1a370 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd567fc640 1 -- 172.21.15.78:0/487199268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd48037070 con 0x7fcd581081b0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd567fc640 1 -- 172.21.15.78:0/487199268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd48032920 con 0x7fcd581081b0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd567fc640 1 -- 172.21.15.78:0/487199268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd48030810 con 0x7fcd581081b0 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/487199268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 msgr2=0x7fcd5810a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:44.170 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/487199268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009820 tx=0x7fcd4802a270 comp rx=0 tx=0).stop 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/487199268 shutdown_connections 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/487199268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5810a5a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/487199268 >> 172.21.15.78:0/487199268 conn(0x7fcd58101db0 msgr2=0x7fcd581041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/487199268 shutdown_connections 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.244+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/487199268 wait complete. 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd5e0c7640 1 Processor -- start 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd5e0c7640 1 -- start start 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd5e0c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd5e0c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810bb60 con 0x7fcd581081b0 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46188/0 (socket says 172.21.15.78:46188) 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.245+0000 7fcd577fe640 1 -- 172.21.15.78:0/771190798 learned_addr learned my addr 172.21.15.78:0/771190798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd577fe640 1 -- 172.21.15.78:0/771190798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5819bcc0 con 0x7fcd581081b0 2024-09-16T01:01:44.171 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd577fe640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd480097f0 tx=0x7fcd480327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd48037040 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd48030c40 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5819bf50 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd4803a5d0 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.246+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5819ec40 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.247+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fcd4803b3f0 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.247+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd58108610 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.247+0000 7fcd54ff9640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 0x7fcd4003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.247+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcd4806c910 con 0x7fcd581081b0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.247+0000 7fcd56ffd640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 0x7fcd4003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:44.172 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.248+0000 7fcd56ffd640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 0x7fcd4003b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0098c0 tx=0x7fcd4c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.252+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcd480313a0 con 0x7fcd581081b0 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.421+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fcd5819f230 con 0x7fcd581081b0 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.422+0000 7fcd54ff9640 1 -- 172.21.15.78:0/771190798 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fcd48038250 con 0x7fcd581081b0 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 msgr2=0x7fcd4003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 0x7fcd4003b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0098c0 tx=0x7fcd4c006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 msgr2=0x7fcd5819b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd480097f0 tx=0x7fcd480327b0 comp rx=0 tx=0).stop 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 shutdown_connections 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcd40039200 0x7fcd4003b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 --2- 172.21.15.78:0/771190798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd581081b0 0x7fcd5819b780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 >> 172.21.15.78:0/771190798 conn(0x7fcd58101db0 msgr2=0x7fcd581029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 shutdown_connections 2024-09-16T01:01:44.173 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:43.424+0000 7fcd5e0c7640 1 -- 172.21.15.78:0/771190798 wait complete. 2024-09-16T01:01:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-16T01:01:44.177 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-09-16T01:01:44.184 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-16T01:01:44.184 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-09-16T01:01:44.184 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-09-16T01:01:44.185 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-16T01:01:44.186 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: Password: 01eddtf558 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:44.187 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config directory 2024-09-16T01:01:44.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/771190798' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-16T01:01:45.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:44 smithi078 ceph-mon[30937]: mgrmap e13: x(active, since 6s) 2024-09-16T01:01:45.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:44 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1417916583' entity='client.admin' 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f670832d640 1 Processor -- start 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f670832d640 1 -- start start 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f670832d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f670832d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6700108170 con 0x7f67001077a0 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f67060c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:45.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f67060c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58568/0 (socket says 172.21.15.78:58568) 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.620+0000 7f67060c9640 1 -- 172.21.15.78:0/580833858 learned_addr learned my addr 172.21.15.78:0/580833858 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.621+0000 7f67060c9640 1 -- 172.21.15.78:0/580833858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6700108950 con 0x7f67001077a0 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.621+0000 7f67060c9640 1 --2- 172.21.15.78:0/580833858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009820 tx=0x7f66fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5c872741e9ac3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.621+0000 7f67050c7640 1 -- 172.21.15.78:0/580833858 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66fc037070 con 0x7f67001077a0 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.621+0000 7f67050c7640 1 -- 172.21.15.78:0/580833858 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66fc032920 con 0x7f67001077a0 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.621+0000 7f67050c7640 1 -- 172.21.15.78:0/580833858 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66fc030810 con 0x7f67001077a0 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 -- 172.21.15.78:0/580833858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 msgr2=0x7f6700107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 --2- 172.21.15.78:0/580833858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009820 tx=0x7f66fc02a270 comp rx=0 tx=0).stop 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 -- 172.21.15.78:0/580833858 shutdown_connections 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 --2- 172.21.15.78:0/580833858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f6700107ba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 -- 172.21.15.78:0/580833858 >> 172.21.15.78:0/580833858 conn(0x7f6700102fd0 msgr2=0x7f67001053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 -- 172.21.15.78:0/580833858 shutdown_connections 2024-09-16T01:01:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.622+0000 7f670832d640 1 -- 172.21.15.78:0/580833858 wait complete. 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f670832d640 1 Processor -- start 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f670832d640 1 -- start start 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f670832d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f670832d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f670010bd00 con 0x7f67001077a0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f67060c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f67060c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58582/0 (socket says 172.21.15.78:58582) 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f67060c9640 1 -- 172.21.15.78:0/1417916583 learned_addr learned my addr 172.21.15.78:0/1417916583 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f67060c9640 1 -- 172.21.15.78:0/1417916583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670019bcb0 con 0x7f67001077a0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.623+0000 7f67060c9640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f66fc030560 tx=0x7f66fc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66fc037040 con 0x7f67001077a0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66fc040070 con 0x7f67001077a0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670019bf40 con 0x7f67001077a0 2024-09-16T01:01:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66fc03a740 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f670019ec30 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.624+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f66fc03b420 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.625+0000 7f66f77fe640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 0x7f66ec03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.625+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670010be80 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.625+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f66fc06ca60 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.625+0000 7f67058c8640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 0x7f66ec03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.625+0000 7f67058c8640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 0x7f66ec03b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098c0 tx=0x7f66f0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.629+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f66fc042940 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.884+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f670019ee00 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.886+0000 7f66f77fe640 1 -- 172.21.15.78:0/1417916583 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f66fc04a200 con 0x7f67001077a0 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 msgr2=0x7f66ec03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:45.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 0x7f66ec03b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098c0 tx=0x7f66f0006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 msgr2=0x7f670019b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f66fc030560 tx=0x7f66fc032670 comp rx=0 tx=0).stop 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 shutdown_connections 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ec0392a0 0x7f66ec03b760 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 --2- 172.21.15.78:0/1417916583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67001077a0 0x7f670019b770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 >> 172.21.15.78:0/1417916583 conn(0x7f6700102fd0 msgr2=0x7f6700103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 shutdown_connections 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-09-16T01:01:44.888+0000 7f670832d640 1 -- 172.21.15.78:0/1417916583 wait complete. 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.464 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:45.465 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2024-09-16T01:01:45.501 INFO:tasks.cephadm:Fetching config... 2024-09-16T01:01:45.502 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:01:45.502 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-16T01:01:45.518 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-16T01:01:45.518 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:01:45.518 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-16T01:01:45.574 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-16T01:01:45.574 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:01:45.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-16T01:01:45.640 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-16T01:01:45.641 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:01:45.641 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-16T01:01:45.695 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-16T01:01:45.696 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T01:01:45.771 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:45.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T01:01:45.825 INFO:teuthology.orchestra.run.smithi148.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:45.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-16T01:01:45.880 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs965fyG4P0NQXhZc6fgyXSI8T5QsEk4hw290jtJD/X54+ZGPkhFz/hBG6/pRJ/EMtLMf1r0pVwZOQgfPE1JwawXpsKSGaImpuPOr7k84claRQg853u7CFLopzCQ5um6MDrV0viTdJ/YhQwh/VBmN0/p+nOHS6THee02GqyZWzFk6jWPzE0yA+AEg/F9Ozta8zYYGtAT0M1/nOveMXFWksOfDE8+e0v5PjZXNIbU84UCUhaygRS9fDbY9quX4oz9Gf/vgHPUKCa7+zfJi7HFLRJSr29AmJiEQxUkJVFgYsU7OVP1156/yXaZ+xr7Q0cR96/gpNUm6sREerld3W2UIaTRLxhrxFummbPYWishMfi9yMefqXvxcH764uxB3tYeFD/iYT3HkBUM500lFrNs091tmKDcBWnzja9Aby6de+mVEQQwGSD/qoY99ckn5+laJG1emCVMSZnH7Ysa2GM4k6W9JfGKovLpaDhP7PhTbBsEvUwtPdPh9NY5RJy+hFWtc= ceph-fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:01:45.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-16T01:01:46.156 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:01:46.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 -- 172.21.15.78:0/1351976254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 msgr2=0x7f965c104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:46.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 --2- 172.21.15.78:0/1351976254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c104490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f964c009880 tx=0x7f964c02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 -- 172.21.15.78:0/1351976254 shutdown_connections 2024-09-16T01:01:46.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 --2- 172.21.15.78:0/1351976254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c104490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 -- 172.21.15.78:0/1351976254 >> 172.21.15.78:0/1351976254 conn(0x7f965c0fbc80 msgr2=0x7f965c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 -- 172.21.15.78:0/1351976254 shutdown_connections 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.618+0000 7f9660aa6640 1 -- 172.21.15.78:0/1351976254 wait complete. 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f9660aa6640 1 Processor -- start 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f9660aa6640 1 -- start start 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f9660aa6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f9660aa6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1076b0 con 0x7f965c1020a0 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f965a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f965a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58596/0 (socket says 172.21.15.78:58596) 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.619+0000 7f965a59c640 1 -- 172.21.15.78:0/532788026 learned_addr learned my addr 172.21.15.78:0/532788026 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f965a59c640 1 -- 172.21.15.78:0/532788026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c101e60 con 0x7f965c1020a0 2024-09-16T01:01:46.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f965a59c640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f964c004f60 tx=0x7f964c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964c03f070 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c100050 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c003ad0 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964c035030 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.620+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c100590 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.621+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f964c042070 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.621+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965c107e30 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.621+0000 7f964b7fe640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 0x7f964403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.621+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f964c06d170 con 0x7f965c1020a0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.622+0000 7f9659d9b640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 0x7f964403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.622+0000 7f9659d9b640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 0x7f964403b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f96500098c0 tx=0x7f9650006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.626+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f964c039350 con 0x7f965c1020a0 2024-09-16T01:01:46.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.799+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f965c190210 con 0x7f965c1020a0 2024-09-16T01:01:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.802+0000 7f964b7fe640 1 -- 172.21.15.78:0/532788026 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f964c003c80 con 0x7f965c1020a0 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 msgr2=0x7f964403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 0x7f964403b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f96500098c0 tx=0x7f9650006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 msgr2=0x7f965c101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f964c004f60 tx=0x7f964c0048f0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 shutdown_connections 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f96440392a0 0x7f964403b760 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 --2- 172.21.15.78:0/532788026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f965c1020a0 0x7f965c101920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 >> 172.21.15.78:0/532788026 conn(0x7f965c0fbc80 msgr2=0x7f965c0fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:46.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 shutdown_connections 2024-09-16T01:01:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:46.804+0000 7f9660aa6640 1 -- 172.21.15.78:0/532788026 wait complete. 2024-09-16T01:01:47.417 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-16T01:01:47.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-16T01:01:48.087 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:47 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/532788026' entity='client.admin' 2024-09-16T01:01:48.393 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:01:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.858+0000 7f91390d2640 1 -- 172.21.15.78:0/408370146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 msgr2=0x7f91341044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 --2- 172.21.15.78:0/408370146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f91341044c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9124009880 tx=0x7f912402a5d0 comp rx=0 tx=0).stop 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 -- 172.21.15.78:0/408370146 shutdown_connections 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 --2- 172.21.15.78:0/408370146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f91341044c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 -- 172.21.15.78:0/408370146 >> 172.21.15.78:0/408370146 conn(0x7f91340fbcd0 msgr2=0x7f91340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 -- 172.21.15.78:0/408370146 shutdown_connections 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.859+0000 7f91390d2640 1 -- 172.21.15.78:0/408370146 wait complete. 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f91390d2640 1 Processor -- start 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f91390d2640 1 -- start start 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f91390d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f91390d2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9134107670 con 0x7f91341020d0 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f9132d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f9132d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58616/0 (socket says 172.21.15.78:58616) 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f9132d9d640 1 -- 172.21.15.78:0/401814677 learned_addr learned my addr 172.21.15.78:0/401814677 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:48.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.860+0000 7f9132d9d640 1 -- 172.21.15.78:0/401814677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9134197280 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f9132d9d640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9124004f60 tx=0x7f91240048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f912403f070 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9124003ad0 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9134197570 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9124035030 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.861+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913419a200 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.862+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f9124042070 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.862+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9134107df0 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.862+0000 7f911bfff640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 0x7f911c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.862+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f912406d0b0 con 0x7f91341020d0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.862+0000 7f913259c640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 0x7f911c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:48.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.863+0000 7f913259c640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 0x7f911c03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f91280098c0 tx=0x7f9128006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:48.867+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9124039350 con 0x7f91341020d0 2024-09-16T01:01:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.043+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f913402d9a0 con 0x7f911c0392a0 2024-09-16T01:01:49.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.046+0000 7f911bfff640 1 -- 172.21.15.78:0/401814677 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f913402d9a0 con 0x7f911c0392a0 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 msgr2=0x7f911c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 0x7f911c03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f91280098c0 tx=0x7f9128006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 msgr2=0x7f9134196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9124004f60 tx=0x7f91240048f0 comp rx=0 tx=0).stop 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 shutdown_connections 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f911c0392a0 0x7f911c03b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 --2- 172.21.15.78:0/401814677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91341020d0 0x7f9134196d40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.048+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 >> 172.21.15.78:0/401814677 conn(0x7f91340fbcd0 msgr2=0x7f91340fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.049+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 shutdown_connections 2024-09-16T01:01:49.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:49.049+0000 7f91390d2640 1 -- 172.21.15.78:0/401814677 wait complete. 2024-09-16T01:01:50.154 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi148 2024-09-16T01:01:50.154 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T01:01:50.154 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.conf 2024-09-16T01:01:50.175 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T01:01:50.175 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:01:50.234 INFO:tasks.cephadm:Adding host smithi148 to orchestrator... 2024-09-16T01:01:50.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch host add smithi148 2024-09-16T01:01:50.491 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:01:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:50 smithi078 ceph-mon[30937]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 -- 172.21.15.78:0/2284147483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 msgr2=0x7fc32c106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 --2- 172.21.15.78:0/2284147483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c106940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc320009880 tx=0x7fc32002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 -- 172.21.15.78:0/2284147483 shutdown_connections 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 --2- 172.21.15.78:0/2284147483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c106940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 -- 172.21.15.78:0/2284147483 >> 172.21.15.78:0/2284147483 conn(0x7fc32c0fc1b0 msgr2=0x7fc32c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 -- 172.21.15.78:0/2284147483 shutdown_connections 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.000+0000 7fc332bec640 1 -- 172.21.15.78:0/2284147483 wait complete. 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc332bec640 1 Processor -- start 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc332bec640 1 -- start start 2024-09-16T01:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc332bec640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc332bec640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10ab90 con 0x7fc32c106560 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc330988640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc330988640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58622/0 (socket says 172.21.15.78:58622) 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc330988640 1 -- 172.21.15.78:0/1119978529 learned_addr learned my addr 172.21.15.78:0/1119978529 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc330988640 1 -- 172.21.15.78:0/1119978529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c198500 con 0x7fc32c106560 2024-09-16T01:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc330988640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc320009230 tx=0x7fc320003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:51.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc32003f070 con 0x7fc32c106560 2024-09-16T01:01:51.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32003bc80 con 0x7fc32c106560 2024-09-16T01:01:51.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc320035030 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.001+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c198790 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.002+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c198bf0 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.002+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc320042070 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.003+0000 7fc329ffb640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 0x7fc31c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.003+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc32006cfd0 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.003+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32c101bd0 con 0x7fc32c106560 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.003+0000 7fc32bfff640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 0x7fc31c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:51.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.003+0000 7fc32bfff640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 0x7fc31c03b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc3180098c0 tx=0x7fc318006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:51.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.008+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc3200372d0 con 0x7fc32c106560 2024-09-16T01:01:51.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:51.176+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}) -- 0x7fc32c198e20 con 0x7fc31c0392a0 2024-09-16T01:01:51.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:01:51.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:01:51.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:51.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:51.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:51.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:51.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:01:51.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:01:51.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:52 smithi078 ceph-mon[30937]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:53.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:53 smithi078 ceph-mon[30937]: Deploying cephadm binary to smithi148 2024-09-16T01:01:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.293+0000 7fc329ffb640 1 -- 172.21.15.78:0/1119978529 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc32c198e20 con 0x7fc31c0392a0 2024-09-16T01:01:54.294 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi148' with addr '172.21.15.148' 2024-09-16T01:01:54.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 msgr2=0x7fc31c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:54.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 0x7fc31c03b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc3180098c0 tx=0x7fc318006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 msgr2=0x7fc32c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc320009230 tx=0x7fc320003d40 comp rx=0 tx=0).stop 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 shutdown_connections 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc31c0392a0 0x7fc31c03b760 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 --2- 172.21.15.78:0/1119978529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc32c106560 0x7fc32c197fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 >> 172.21.15.78:0/1119978529 conn(0x7fc32c0fc1b0 msgr2=0x7fc32c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 shutdown_connections 2024-09-16T01:01:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:54.295+0000 7fc332bec640 1 -- 172.21.15.78:0/1119978529 wait complete. 2024-09-16T01:01:54.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T01:01:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:01:55.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:55.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:55 smithi078 ceph-mon[30937]: Added host smithi148 2024-09-16T01:01:55.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:01:55.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:55.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.643+0000 7fcbe324f640 1 -- 172.21.15.78:0/326254783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 msgr2=0x7fcbdc106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.643+0000 7fcbe324f640 1 --2- 172.21.15.78:0/326254783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc106920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009810 tx=0x7fcbd002a540 comp rx=0 tx=0).stop 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.643+0000 7fcbe324f640 1 -- 172.21.15.78:0/326254783 shutdown_connections 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.643+0000 7fcbe324f640 1 --2- 172.21.15.78:0/326254783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc106920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.643+0000 7fcbe324f640 1 -- 172.21.15.78:0/326254783 >> 172.21.15.78:0/326254783 conn(0x7fcbdc077310 msgr2=0x7fcbdc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 -- 172.21.15.78:0/326254783 shutdown_connections 2024-09-16T01:01:55.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 -- 172.21.15.78:0/326254783 wait complete. 2024-09-16T01:01:55.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 Processor -- start 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 -- start start 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe324f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10ab70 con 0x7fcbdc106540 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe0feb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.644+0000 7fcbe0feb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38330/0 (socket says 172.21.15.78:38330) 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbe0feb640 1 -- 172.21.15.78:0/3587973438 learned_addr learned my addr 172.21.15.78:0/3587973438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbe0feb640 1 -- 172.21.15.78:0/3587973438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc103d40 con 0x7fcbdc106540 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbe0feb640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00045e0 tx=0x7fcbd0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:55.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd0030e70 con 0x7fcbdc106540 2024-09-16T01:01:55.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbd00318e0 con 0x7fcbdc106540 2024-09-16T01:01:55.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc103fd0 con 0x7fcbdc106540 2024-09-16T01:01:55.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd00049c0 con 0x7fcbdc106540 2024-09-16T01:01:55.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.645+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc102810 con 0x7fcbdc106540 2024-09-16T01:01:55.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.646+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fcbd0039040 con 0x7fcbdc106540 2024-09-16T01:01:55.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.646+0000 7fcbd9ffb640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 0x7fcbcc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:55.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.646+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcbd006e590 con 0x7fcbdc106540 2024-09-16T01:01:55.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.646+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc101bb0 con 0x7fcbdc106540 2024-09-16T01:01:55.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.647+0000 7fcbdbfff640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 0x7fcbcc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:55.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.647+0000 7fcbdbfff640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 0x7fcbcc03b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098c0 tx=0x7fcbc8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:55.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.651+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcbd0040030 con 0x7fcbdc106540 2024-09-16T01:01:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.822+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbdc103630 con 0x7fcbcc0392a0 2024-09-16T01:01:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.823+0000 7fcbd9ffb640 1 -- 172.21.15.78:0/3587973438 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fcbdc103630 con 0x7fcbcc0392a0 2024-09-16T01:01:55.824 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:01:55.824 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}] 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 msgr2=0x7fcbcc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 0x7fcbcc03b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098c0 tx=0x7fcbc8006db0 comp rx=0 tx=0).stop 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 msgr2=0x7fcbdc102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00045e0 tx=0x7fcbd0004610 comp rx=0 tx=0).stop 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 shutdown_connections 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcbcc0392a0 0x7fcbcc03b760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 --2- 172.21.15.78:0/3587973438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc106540 0x7fcbdc102100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 >> 172.21.15.78:0/3587973438 conn(0x7fcbdc077310 msgr2=0x7fcbdc0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 shutdown_connections 2024-09-16T01:01:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:55.825+0000 7fcbe324f640 1 -- 172.21.15.78:0/3587973438 wait complete. 2024-09-16T01:01:56.478 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-09-16T01:01:56.478 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T01:01:56.478 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-09-16T01:01:56.499 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T01:01:56.499 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:01:56.557 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-09-16T01:01:56.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch host add smithi193 2024-09-16T01:01:56.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:56.707 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:56 smithi078 ceph-mon[30937]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:01:56.801 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:01:57.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- 172.21.15.78:0/182096673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 msgr2=0x7fd288106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:01:57.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 --2- 172.21.15.78:0/182096673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288106920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd278009810 tx=0x7fd27802a540 comp rx=0 tx=0).stop 2024-09-16T01:01:57.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- 172.21.15.78:0/182096673 shutdown_connections 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 --2- 172.21.15.78:0/182096673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288106920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- 172.21.15.78:0/182096673 >> 172.21.15.78:0/182096673 conn(0x7fd288077310 msgr2=0x7fd2880fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- 172.21.15.78:0/182096673 shutdown_connections 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- 172.21.15.78:0/182096673 wait complete. 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 Processor -- start 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.251+0000 7fd28db26640 1 -- start start 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd28db26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd28db26640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd28810ab70 con 0x7fd288106540 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38350/0 (socket says 172.21.15.78:38350) 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2877fe640 1 -- 172.21.15.78:0/1062536258 learned_addr learned my addr 172.21.15.78:0/1062536258 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:01:57.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2877fe640 1 -- 172.21.15.78:0/1062536258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd288198510 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2877fe640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2780045e0 tx=0x7fd278004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd278030e70 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2881987a0 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2780318e0 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2780049c0 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.252+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd288198c00 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.253+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fd278039040 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.253+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd288101bb0 con 0x7fd288106540 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.254+0000 7fd2857fa640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 0x7fd27403b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:01:57.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.254+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd27806d560 con 0x7fd288106540 2024-09-16T01:01:57.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.254+0000 7fd27ffff640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 0x7fd27403b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:01:57.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.254+0000 7fd27ffff640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 0x7fd27403b7c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd2700098c0 tx=0x7fd270006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:01:57.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.259+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd278040030 con 0x7fd288106540 2024-09-16T01:01:57.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:01:57.430+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) -- 0x7fd2881069a0 con 0x7fd274039300 2024-09-16T01:01:59.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:01:59.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:01:59.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi148", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:01:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:02:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: Deploying cephadm binary to smithi193 2024-09-16T01:02:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:01:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:00.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.838+0000 7fd2857fa640 1 -- 172.21.15.78:0/1062536258 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd2881069a0 con 0x7fd274039300 2024-09-16T01:02:00.840 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 msgr2=0x7fd27403b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 0x7fd27403b7c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd2700098c0 tx=0x7fd270006db0 comp rx=0 tx=0).stop 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 msgr2=0x7fd288197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2780045e0 tx=0x7fd278004610 comp rx=0 tx=0).stop 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 shutdown_connections 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd274039300 0x7fd27403b7c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 --2- 172.21.15.78:0/1062536258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288106540 0x7fd288197fd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:00.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 >> 172.21.15.78:0/1062536258 conn(0x7fd288077310 msgr2=0x7fd2880fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:00.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 shutdown_connections 2024-09-16T01:02:00.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:00.841+0000 7fd28db26640 1 -- 172.21.15.78:0/1062536258 wait complete. 2024-09-16T01:02:01.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:00 smithi078 ceph-mon[30937]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:02:01.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:00 smithi078 ceph-mon[30937]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:01.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:01.264 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:01.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-16T01:02:01.782 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:02:02.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:01 smithi078 ceph-mon[30937]: Added host smithi193 2024-09-16T01:02:02.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:02.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 -- 172.21.15.78:0/556885716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 msgr2=0x7f3210102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 --2- 172.21.15.78:0/556885716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f3210102bd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3208009880 tx=0x7f320802a670 comp rx=0 tx=0).stop 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 -- 172.21.15.78:0/556885716 shutdown_connections 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 --2- 172.21.15.78:0/556885716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f3210102bd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 -- 172.21.15.78:0/556885716 >> 172.21.15.78:0/556885716 conn(0x7f32100fc1b0 msgr2=0x7f32100fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 -- 172.21.15.78:0/556885716 shutdown_connections 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.227+0000 7f32175af640 1 -- 172.21.15.78:0/556885716 wait complete. 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f32175af640 1 Processor -- start 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f32175af640 1 -- start start 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f32175af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f32175af640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f321010bcc0 con 0x7f32101027f0 2024-09-16T01:02:02.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f321534b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:02.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f321534b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38362/0 (socket says 172.21.15.78:38362) 2024-09-16T01:02:02.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f321534b640 1 -- 172.21.15.78:0/2332092427 learned_addr learned my addr 172.21.15.78:0/2332092427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.228+0000 7f321534b640 1 -- 172.21.15.78:0/2332092427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3210197710 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f321534b640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3208009230 tx=0x7f3208003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f320803f070 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32101979a0 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f320803bc80 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3208035030 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.229+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3210197e00 con 0x7f32101027f0 2024-09-16T01:02:02.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f3208042070 con 0x7f32101027f0 2024-09-16T01:02:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f320e7fc640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 0x7f31f803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321006a710 con 0x7f32101027f0 2024-09-16T01:02:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f320806cc80 con 0x7f32101027f0 2024-09-16T01:02:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f3214b4a640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 0x7f31f803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:02.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.230+0000 7f3214b4a640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 0x7f31f803b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f32040098c0 tx=0x7f3204006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:02.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.235+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3208031e20 con 0x7f32101027f0 2024-09-16T01:02:02.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.411+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3210198530 con 0x7f31f80392a0 2024-09-16T01:02:02.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.412+0000 7f320e7fc640 1 -- 172.21.15.78:0/2332092427 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f3210198530 con 0x7f31f80392a0 2024-09-16T01:02:02.412 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:02:02.413 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 msgr2=0x7f31f803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 0x7f31f803b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f32040098c0 tx=0x7f3204006db0 comp rx=0 tx=0).stop 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 msgr2=0x7f32101971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3208009230 tx=0x7f3208003d40 comp rx=0 tx=0).stop 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 shutdown_connections 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f31f80392a0 0x7f31f803b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 --2- 172.21.15.78:0/2332092427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32101027f0 0x7f32101971d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 >> 172.21.15.78:0/2332092427 conn(0x7f32100fc1b0 msgr2=0x7f32101055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 shutdown_connections 2024-09-16T01:02:02.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:02.414+0000 7f32175af640 1 -- 172.21.15.78:0/2332092427 wait complete. 2024-09-16T01:02:03.000 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-16T01:02:03.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-16T01:02:03.241 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:02:03.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:02 smithi078 ceph-mon[30937]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:03.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:03.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 -- 172.21.15.78:0/1856490581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 msgr2=0x7f7274107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:03.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 --2- 172.21.15.78:0/1856490581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f7274107a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7270009880 tx=0x7f727002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:02:03.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 -- 172.21.15.78:0/1856490581 shutdown_connections 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 --2- 172.21.15.78:0/1856490581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f7274107a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 -- 172.21.15.78:0/1856490581 >> 172.21.15.78:0/1856490581 conn(0x7f72740fd290 msgr2=0x7f72740ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 -- 172.21.15.78:0/1856490581 shutdown_connections 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.712+0000 7f727abb4640 1 -- 172.21.15.78:0/1856490581 wait complete. 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f727abb4640 1 Processor -- start 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f727abb4640 1 -- start start 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f727abb4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f727abb4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f727410bc90 con 0x7f7274107660 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f7278950640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f7278950640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38382/0 (socket says 172.21.15.78:38382) 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.713+0000 7f7278950640 1 -- 172.21.15.78:0/2212069049 learned_addr learned my addr 172.21.15.78:0/2212069049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f7278950640 1 -- 172.21.15.78:0/2212069049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274197600 con 0x7f7274107660 2024-09-16T01:02:03.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f7278950640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7270009230 tx=0x7f7270003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:03.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f727003f070 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f727003bd40 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7270035030 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7274197890 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.714+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727419b990 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.715+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7270042070 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.715+0000 7f7269ffb640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 0x7f725c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.715+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7274102cd0 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.715+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f727006cf50 con 0x7f7274107660 2024-09-16T01:02:03.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.715+0000 7f726bfff640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 0x7f725c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:03.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.716+0000 7f726bfff640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 0x7f725c03b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72640098c0 tx=0x7f7264006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:03.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.720+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7270030d60 con 0x7f7274107660 2024-09-16T01:02:03.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.889+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7274198420 con 0x7f7274107660 2024-09-16T01:02:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.976+0000 7f7269ffb640 1 -- 172.21.15.78:0/2212069049 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f727003b860 con 0x7f7274107660 2024-09-16T01:02:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 msgr2=0x7f725c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 0x7f725c03b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72640098c0 tx=0x7f7264006db0 comp rx=0 tx=0).stop 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 msgr2=0x7f72741970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7270009230 tx=0x7f7270003d80 comp rx=0 tx=0).stop 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 shutdown_connections 2024-09-16T01:02:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f725c0392a0 0x7f725c03b760 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 --2- 172.21.15.78:0/2212069049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7274107660 0x7f72741970c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 >> 172.21.15.78:0/2212069049 conn(0x7f72740fd290 msgr2=0x7f72740febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 shutdown_connections 2024-09-16T01:02:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:03.979+0000 7f727abb4640 1 -- 172.21.15.78:0/2212069049 wait complete. 2024-09-16T01:02:04.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:03 smithi078 ceph-mon[30937]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:02:04.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:03 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2212069049' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-16T01:02:04.598 INFO:tasks.cephadm:Adding mon.a on smithi078 2024-09-16T01:02:04.598 INFO:tasks.cephadm:Adding mon.b on smithi148 2024-09-16T01:02:04.598 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-09-16T01:02:04.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi148:172.21.15.148=b;smithi193:172.21.15.193=c' 2024-09-16T01:02:04.871 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T01:02:04.917 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2212069049' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:02:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:05.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 -- 172.21.15.193:0/2928011183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 msgr2=0x7fafe81044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:05.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 --2- 172.21.15.193:0/2928011183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe81044c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009880 tx=0x7fafd802a5d0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 -- 172.21.15.193:0/2928011183 shutdown_connections 2024-09-16T01:02:05.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 --2- 172.21.15.193:0/2928011183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe81044c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.459 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 -- 172.21.15.193:0/2928011183 >> 172.21.15.193:0/2928011183 conn(0x7fafe80fbcd0 msgr2=0x7fafe80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 -- 172.21.15.193:0/2928011183 shutdown_connections 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 -- 172.21.15.193:0/2928011183 wait complete. 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.434+0000 7fafecb32640 1 Processor -- start 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafecb32640 1 -- start start 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafecb32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafecb32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8107670 con 0x7fafe81020d0 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafe659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:05.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafe659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.193:57164/0 (socket says 172.21.15.193:57164) 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafe659c640 1 -- 172.21.15.193:0/1643516856 learned_addr learned my addr 172.21.15.193:0/1643516856 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.435+0000 7fafe659c640 1 -- 172.21.15.193:0/1643516856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe8101e90 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafe659c640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafd8004f60 tx=0x7fafd80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd803f070 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd8003ad0 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe8100080 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafd8035030 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.436+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe81005c0 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.437+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fafd8042070 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.437+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe8107df0 con 0x7fafe81020d0 2024-09-16T01:02:05.461 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.437+0000 7fafd77fe640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 0x7fafd003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:05.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.437+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fafd806d170 con 0x7fafe81020d0 2024-09-16T01:02:05.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.437+0000 7fafe5d9b640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 0x7fafd003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:05.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.438+0000 7fafe5d9b640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 0x7fafd003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0098c0 tx=0x7fafdc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:05.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.442+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fafd8039350 con 0x7fafe81020d0 2024-09-16T01:02:05.634 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.632+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi148:172.21.15.148=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) -- 0x7fafe8190410 con 0x7fafd00392a0 2024-09-16T01:02:05.640 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.638+0000 7fafd77fe640 1 -- 172.21.15.193:0/1643516856 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fafe8190410 con 0x7fafd00392a0 2024-09-16T01:02:05.640 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-09-16T01:02:05.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 msgr2=0x7fafd003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:05.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 0x7fafd003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fafdc0098c0 tx=0x7fafdc006db0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 msgr2=0x7fafe8101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:05.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafd8004f60 tx=0x7fafd80048f0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 shutdown_connections 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fafd00392a0 0x7fafd003b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 --2- 172.21.15.193:0/1643516856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafe81020d0 0x7fafe8101950 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 >> 172.21.15.193:0/1643516856 conn(0x7fafe80fbcd0 msgr2=0x7fafe80fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 shutdown_connections 2024-09-16T01:02:05.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:05.640+0000 7fafecb32640 1 -- 172.21.15.193:0/1643516856 wait complete. 2024-09-16T01:02:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:02:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:06.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:06.284 DEBUG:teuthology.orchestra.run.smithi148:mon.b> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.b.service 2024-09-16T01:02:06.287 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.c.service 2024-09-16T01:02:06.289 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T01:02:06.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T01:02:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:02:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:02:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi148:172.21.15.148=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi148:172.21.15.148=b;smithi193:172.21.15.193=c;count:3 2024-09-16T01:02:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: Deploying daemon mon.c on smithi193 2024-09-16T01:02:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:06 smithi078 ceph-mon[30937]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:07.472 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 -- 172.21.15.193:0/1061102760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 msgr2=0x7f8260106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 --2- 172.21.15.193:0/1061102760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260106940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8250009880 tx=0x7f825002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 -- 172.21.15.193:0/1061102760 shutdown_connections 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 --2- 172.21.15.193:0/1061102760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260106940 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 -- 172.21.15.193:0/1061102760 >> 172.21.15.193:0/1061102760 conn(0x7f82600fc1b0 msgr2=0x7f82600fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 -- 172.21.15.193:0/1061102760 shutdown_connections 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.109+0000 7f82654b4640 1 -- 172.21.15.193:0/1061102760 wait complete. 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f82654b4640 1 Processor -- start 2024-09-16T01:02:08.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f82654b4640 1 -- start start 2024-09-16T01:02:08.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f82654b4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:08.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f82654b4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f826010ab90 con 0x7f8260106560 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f825effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f825effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.193:57184/0 (socket says 172.21.15.193:57184) 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.110+0000 7f825effd640 1 -- 172.21.15.193:0/731493870 learned_addr learned my addr 172.21.15.193:0/731493870 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f825effd640 1 -- 172.21.15.193:0/731493870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8260199840 con 0x7f8260106560 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f825effd640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8250009230 tx=0x7f8250003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f825003f070 con 0x7f8260106560 2024-09-16T01:02:08.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825003bc80 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8260199ad0 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8250035030 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.111+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826019dbd0 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.112+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f8250042070 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.112+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8260101bd0 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.112+0000 7f8247fff640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 0x7f824803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.112+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f825006d030 con 0x7f8260106560 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.112+0000 7f825e7fc640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 0x7f824803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:08.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.113+0000 7f825e7fc640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 0x7f824803b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f82540098c0 tx=0x7f8254006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:08.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.117+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f825004c350 con 0x7f8260106560 2024-09-16T01:02:08.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.352+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f826019a660 con 0x7f8260106560 2024-09-16T01:02:08.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.353+0000 7f8247fff640 1 -- 172.21.15.193:0/731493870 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f8250031e20 con 0x7f8260106560 2024-09-16T01:02:08.355 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:02:08.355 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","modified":"2024-09-16T01:00:42.542629Z","created":"2024-09-16T01:00:42.542629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-16T01:02:08.355 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-09-16T01:02:08.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 msgr2=0x7f824803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:08.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 0x7f824803b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f82540098c0 tx=0x7f8254006db0 comp rx=0 tx=0).stop 2024-09-16T01:02:08.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 msgr2=0x7f8260199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:08.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8250009230 tx=0x7f8250003d40 comp rx=0 tx=0).stop 2024-09-16T01:02:08.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 shutdown_connections 2024-09-16T01:02:08.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82480392a0 0x7f824803b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:08.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 --2- 172.21.15.193:0/731493870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8260106560 0x7f8260199300 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:08.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 >> 172.21.15.193:0/731493870 conn(0x7f82600fc1b0 msgr2=0x7f82600fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:08.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 shutdown_connections 2024-09-16T01:02:08.358 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:08.355+0000 7f82654b4640 1 -- 172.21.15.193:0/731493870 wait complete. 2024-09-16T01:02:09.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:08 smithi078 ceph-mon[30937]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:09.618 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/731493870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:10.620 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T01:02:10.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T01:02:10.644 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:10 smithi193 systemd[1]: Started Ceph mon.c for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:02:10.869 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:02:15.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 -- 172.21.15.193:0/927754760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee64003530 msgr2=0x7fee640059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:15.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 --2- 172.21.15.193:0/927754760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee64003530 0x7fee640059c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fee7c06aa60 tx=0x7fee6c02c490 comp rx=0 tx=0).stop 2024-09-16T01:02:15.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 -- 172.21.15.193:0/927754760 shutdown_connections 2024-09-16T01:02:15.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 --2- 172.21.15.193:0/927754760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee64003530 0x7fee640059c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.695 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 -- 172.21.15.193:0/927754760 >> 172.21.15.193:0/927754760 conn(0x7fee7c0fc220 msgr2=0x7fee7c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:15.696 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 -- 172.21.15.193:0/927754760 shutdown_connections 2024-09-16T01:02:15.696 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.693+0000 7fee828ef640 1 -- 172.21.15.193:0/927754760 wait complete. 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 Processor -- start 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 -- start start 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 0x7fee7c19bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c109780 con 0x7fee7c1978c0 2024-09-16T01:02:15.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee828ef640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c109600 con 0x7fee64003530 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.695+0000 7fee7bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:43284/0 (socket says 172.21.15.193:43284) 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 -- 172.21.15.193:0/1748360494 learned_addr learned my addr 172.21.15.193:0/1748360494 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 -- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 msgr2=0x7fee7c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 -- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 msgr2=0x7fee7c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7b7fe640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 0x7fee7c19bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7bfff640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7b7fe640 1 -- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 msgr2=0x7fee7c197380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7b7fe640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7b7fe640 1 -- 172.21.15.193:0/1748360494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c197ee0 con 0x7fee7c1978c0 2024-09-16T01:02:15.699 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee7b7fe640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 0x7fee7c19bc00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee7000b3d0 tx=0x7fee7000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:15.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.696+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fee70011630 con 0x7fee7c1978c0 2024-09-16T01:02:15.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.697+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c19c140 con 0x7fee7c1978c0 2024-09-16T01:02:15.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.697+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee70011cb0 con 0x7fee7c1978c0 2024-09-16T01:02:15.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.697+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fee70010df0 con 0x7fee7c1978c0 2024-09-16T01:02:15.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.697+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c19c680 con 0x7fee7c1978c0 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.698+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fee70018020 con 0x7fee7c1978c0 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.698+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7c06ad10 con 0x7fee7c1978c0 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.698+0000 7fee797fa640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 0x7fee6803b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.698+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fee7004c070 con 0x7fee7c1978c0 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.698+0000 7fee7bfff640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 0x7fee6803b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:15.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.699+0000 7fee7bfff640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 0x7fee6803b500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee6c012c10 tx=0x7fee6c0323a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:15.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.703+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fee7001f020 con 0x7fee7c1978c0 2024-09-16T01:02:15.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: mon.a calling monitor election 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: mon.c calling monitor election 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: monmap epoch 2 2024-09-16T01:02:15.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: last_changed 2024-09-16T01:02:10.404230+0000 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: created 2024-09-16T01:00:42.542629+0000 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: min_mon_release 19 (squid) 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: election_strategy: 1 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: fsmap 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: mgrmap e13: x(active, since 37s) 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: overall HEALTH_OK 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:15.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:15.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:02:15.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: mon.a calling monitor election 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: mon.c calling monitor election 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: monmap epoch 2 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:15.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: last_changed 2024-09-16T01:02:10.404230+0000 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: created 2024-09-16T01:00:42.542629+0000 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: min_mon_release 19 (squid) 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: election_strategy: 1 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: fsmap 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: mgrmap e13: x(active, since 37s) 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: overall HEALTH_OK 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:15.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:15.799 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:15.940 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.939+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fee7c190f70 con 0x7fee7c1978c0 2024-09-16T01:02:15.941 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.940+0000 7fee797fa640 1 -- 172.21.15.193:0/1748360494 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fee70018310 con 0x7fee7c1978c0 2024-09-16T01:02:15.941 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:02:15.941 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","modified":"2024-09-16T01:02:10.404230Z","created":"2024-09-16T01:00:42.542629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T01:02:15.942 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-09-16T01:02:15.943 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 msgr2=0x7fee6803b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:15.943 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 0x7fee6803b500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fee6c012c10 tx=0x7fee6c0323a0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.943 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 msgr2=0x7fee7c19bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:15.943 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 0x7fee7c19bc00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fee7000b3d0 tx=0x7fee7000b890 comp rx=0 tx=0).stop 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 shutdown_connections 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fee68039040 0x7fee6803b500 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee7c1978c0 0x7fee7c19bc00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 --2- 172.21.15.193:0/1748360494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee64003530 0x7fee7c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 >> 172.21.15.193:0/1748360494 conn(0x7fee7c0fc220 msgr2=0x7fee7c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 shutdown_connections 2024-09-16T01:02:15.944 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:15.942+0000 7fee828ef640 1 -- 172.21.15.193:0/1748360494 wait complete. 2024-09-16T01:02:16.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:16 smithi078 ceph-mon[30937]: Deploying daemon mon.b on smithi148 2024-09-16T01:02:16.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:16 smithi078 ceph-mon[30937]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:16.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:16 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/1748360494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:16.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:16 smithi193 ceph-mon[35701]: Deploying daemon mon.b on smithi148 2024-09-16T01:02:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:16 smithi193 ceph-mon[35701]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:16 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/1748360494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:17.514 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T01:02:17.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T01:02:17.766 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:02:18.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.158+0000 7f0dcdd81640 1 -- 172.21.15.193:0/3089993509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 msgr2=0x7f0dc8107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.158+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/3089993509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc8107a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009880 tx=0x7f0db802a5d0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.158+0000 7f0dcdd81640 1 -- 172.21.15.193:0/3089993509 shutdown_connections 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/3089993509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc8107a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 -- 172.21.15.193:0/3089993509 >> 172.21.15.193:0/3089993509 conn(0x7f0dc80fd320 msgr2=0x7f0dc80ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 -- 172.21.15.193:0/3089993509 shutdown_connections 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 -- 172.21.15.193:0/3089993509 wait complete. 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 Processor -- start 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 -- start start 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.159+0000 7f0dcdd81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dcdd81640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:18.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dcdd81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc810bd20 con 0x7f0dc8107660 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dcdd81640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc810bba0 con 0x7f0dc8074a30 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.193:50106/0 (socket says 172.21.15.193:50106) 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 -- 172.21.15.193:0/4136236088 learned_addr learned my addr 172.21.15.193:0/4136236088 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc6ffd640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:18.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc6ffd640 1 -- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 msgr2=0x7f0dc8074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc6ffd640 1 -- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 msgr2=0x7f0dc8074e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc6ffd640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc6ffd640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 -- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 msgr2=0x7f0dc8074e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 -- 172.21.15.193:0/4136236088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc80763e0 con 0x7f0dc8107660 2024-09-16T01:02:18.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.160+0000 7f0dc77fe640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0db8032c50 tx=0x7f0db8032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:18.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.161+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0db8035070 con 0x7f0dc8107660 2024-09-16T01:02:18.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.161+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc81a35f0 con 0x7f0dc8107660 2024-09-16T01:02:18.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.161+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc81a3b80 con 0x7f0dc8107660 2024-09-16T01:02:18.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.161+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db8003ce0 con 0x7f0dc8107660 2024-09-16T01:02:18.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.161+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0db803ba90 con 0x7f0dc8107660 2024-09-16T01:02:18.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f0db8042030 con 0x7f0dc8107660 2024-09-16T01:02:18.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc8102da0 con 0x7f0dc8107660 2024-09-16T01:02:18.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dc4ff9640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 0x7f0db003b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:18.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0db8040070 con 0x7f0dc8107660 2024-09-16T01:02:18.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dc6ffd640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 0x7f0db003b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:18.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.162+0000 7f0dc6ffd640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 0x7f0db003b500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc006fd0 tx=0x7f0dbc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:18.168 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.167+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0db8038350 con 0x7f0dc8107660 2024-09-16T01:02:18.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.397+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0dc81a3e30 con 0x7f0dc8107660 2024-09-16T01:02:18.399 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.398+0000 7f0dc4ff9640 1 -- 172.21.15.193:0/4136236088 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f0db8040d60 con 0x7f0dc8107660 2024-09-16T01:02:18.399 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:02:18.399 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","modified":"2024-09-16T01:02:10.404230Z","created":"2024-09-16T01:00:42.542629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-16T01:02:18.399 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 msgr2=0x7f0db003b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 0x7f0db003b500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc006fd0 tx=0x7f0dbc008040 comp rx=0 tx=0).stop 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 msgr2=0x7f0dc80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0db8032c50 tx=0x7f0db8032c80 comp rx=0 tx=0).stop 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 shutdown_connections 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0db0039040 0x7f0db003b500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8074a30 0x7f0dc8074e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 --2- 172.21.15.193:0/4136236088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dc8107660 0x7f0dc80744f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:18.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 >> 172.21.15.193:0/4136236088 conn(0x7f0dc80fd320 msgr2=0x7f0dc80fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:18.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 shutdown_connections 2024-09-16T01:02:18.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:18.400+0000 7f0dcdd81640 1 -- 172.21.15.193:0/4136236088 wait complete. 2024-09-16T01:02:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:18 smithi078 ceph-mon[30937]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:18 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/4136236088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:18 smithi193 ceph-mon[35701]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:18 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/4136236088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:19.987 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-16T01:02:19.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-16T01:02:20.241 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:02:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:24 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: mon.a calling monitor election 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: mon.c calling monitor election 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: mon.b calling monitor election 2024-09-16T01:02:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: monmap epoch 3 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: last_changed 2024-09-16T01:02:19.538500+0000 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: created 2024-09-16T01:00:42.542629+0000 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: min_mon_release 19 (squid) 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: election_strategy: 1 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-16T01:02:25.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: 2: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.b 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: fsmap 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: mgrmap e13: x(active, since 46s) 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: overall HEALTH_OK 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.031 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: mon.a calling monitor election 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: mon.c calling monitor election 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: mon.b calling monitor election 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: monmap epoch 3 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: last_changed 2024-09-16T01:02:19.538500+0000 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: created 2024-09-16T01:00:42.542629+0000 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: min_mon_release 19 (squid) 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: election_strategy: 1 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: 0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-09-16T01:02:25.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: 2: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.b 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: fsmap 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: osdmap e4: 0 total, 0 up, 0 in 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: mgrmap e13: x(active, since 46s) 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: overall HEALTH_OK 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 -- 172.21.15.193:0/1906923107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0f0003530 msgr2=0x7fc0f00059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 --2- 172.21.15.193:0/1906923107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0f0003530 0x7fc0f00059c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc10806a1e0 tx=0x7fc0f802c490 comp rx=0 tx=0).stop 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 -- 172.21.15.193:0/1906923107 shutdown_connections 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 --2- 172.21.15.193:0/1906923107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0f0003530 0x7fc0f00059c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 -- 172.21.15.193:0/1906923107 >> 172.21.15.193:0/1906923107 conn(0x7fc1080fc280 msgr2=0x7fc1080fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 -- 172.21.15.193:0/1906923107 shutdown_connections 2024-09-16T01:02:25.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.058+0000 7fc10d07b640 1 -- 172.21.15.193:0/1906923107 wait complete. 2024-09-16T01:02:25.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 Processor -- start 2024-09-16T01:02:25.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 -- start start 2024-09-16T01:02:25.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:25.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 0x7fc10819bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:25.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc10819c240 0x7fc10819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:25.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc108109760 con 0x7fc108197970 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081095e0 con 0x7fc10819c240 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.059+0000 7fc10d07b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1081098e0 con 0x7fc0f0003530 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:59326/0 (socket says 172.21.15.193:59326) 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 -- 172.21.15.193:0/3004486087 learned_addr learned my addr 172.21.15.193:0/3004486087 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 0x7fc10819bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 msgr2=0x7fc108197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 msgr2=0x7fc108197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc106d9d640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:02:25.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10759e640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc10819c240 0x7fc10819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 msgr2=0x7fc108197430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc10819c240 msgr2=0x7fc10819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc10819c240 0x7fc10819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 -- 172.21.15.193:0/3004486087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1081a3fb0 con 0x7fc108197970 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10659c640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 0x7fc10819bd00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b450 tx=0x7fc0fc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc004230 con 0x7fc108197970 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0fc0043d0 con 0x7fc108197970 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc003cb0 con 0x7fc108197970 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.060+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1081a41e0 con 0x7fc108197970 2024-09-16T01:02:25.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.061+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081a4740 con 0x7fc108197970 2024-09-16T01:02:25.064 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.061+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10810a8f0 con 0x7fc108197970 2024-09-16T01:02:25.064 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.062+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc0fc00f050 con 0x7fc108197970 2024-09-16T01:02:25.064 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.062+0000 7fc0effff640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 0x7fc0f403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:25.064 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.062+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc0fc04c110 con 0x7fc108197970 2024-09-16T01:02:25.065 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.062+0000 7fc106d9d640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 0x7fc0f403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:25.065 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.063+0000 7fc106d9d640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 0x7fc0f403bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc0f802ca00 tx=0x7fc0f802c990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:25.067 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.067+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc0fc00d040 con 0x7fc108197970 2024-09-16T01:02:25.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.303+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc1081a4b90 con 0x7fc108197970 2024-09-16T01:02:25.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.304+0000 7fc0effff640 1 -- 172.21.15.193:0/3004486087 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fc0fc003e50 con 0x7fc108197970 2024-09-16T01:02:25.305 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:02:25.305 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","modified":"2024-09-16T01:02:19.538500Z","created":"2024-09-16T01:00:42.542629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-16T01:02:25.306 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-09-16T01:02:25.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 msgr2=0x7fc0f403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 0x7fc0f403bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc0f802ca00 tx=0x7fc0f802c990 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 msgr2=0x7fc10819bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 0x7fc10819bd00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b450 tx=0x7fc0fc00b910 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 shutdown_connections 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc0f4039730 0x7fc0f403bbf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.306+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc10819c240 0x7fc10819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.307+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc108197970 0x7fc10819bd00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.307+0000 7fc10d07b640 1 --2- 172.21.15.193:0/3004486087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0f0003530 0x7fc108197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.307+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 >> 172.21.15.193:0/3004486087 conn(0x7fc1080fc280 msgr2=0x7fc1080fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.307+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 shutdown_connections 2024-09-16T01:02:25.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:25.307+0000 7fc10d07b640 1 -- 172.21.15.193:0/3004486087 wait complete. 2024-09-16T01:02:25.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:25 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/3004486087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:25 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/3004486087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.941 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:25.969 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-16T01:02:25.970 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-16T01:02:25.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:25 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/3004486087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-16T01:02:25.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:02:26.222 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:02:26.673 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:02:26.538+0000 7f3f0eced640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T01:02:26.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/2664171534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38102730 msgr2=0x7fdb38102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:26.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/2664171534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38102730 0x7fdb38102b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009810 tx=0x7fdb2c02a540 comp rx=0 tx=0).stop 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/2664171534 shutdown_connections 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/2664171534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38102730 0x7fdb38102b10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/2664171534 >> 172.21.15.78:0/2664171534 conn(0x7fdb38077310 msgr2=0x7fdb380fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/2664171534 shutdown_connections 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.800+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/2664171534 wait complete. 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 Processor -- start 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 -- start start 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 0x7fdb38197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 0x7fdb3819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810bc30 con 0x7fdb3819c160 2024-09-16T01:02:26.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810bab0 con 0x7fdb38102730 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3f0f9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3810bdb0 con 0x7fdb38197890 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47692/0 (socket says 172.21.15.78:47692) 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 -- 172.21.15.78:0/662465707 learned_addr learned my addr 172.21.15.78:0/662465707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb37fff640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 0x7fdb3819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3ce95640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 0x7fdb38197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 -- 172.21.15.78:0/662465707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 msgr2=0x7fdb3819bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 0x7fdb3819bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 -- 172.21.15.78:0/662465707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 msgr2=0x7fdb38197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 0x7fdb38197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3d696640 1 -- 172.21.15.78:0/662465707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb381a3ed0 con 0x7fdb3819c160 2024-09-16T01:02:26.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb3ce95640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 0x7fdb38197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.801+0000 7fdb37fff640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 0x7fdb3819bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb3d696640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000ba50 tx=0x7fdb3000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb30019070 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3000cbe0 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb300076a0 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb381a4100 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.802+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb381a4610 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fdb3001c070 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb35ffb640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 0x7fdb2803bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb3004bd20 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb38069f90 con 0x7fdb3819c160 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb3ce95640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 0x7fdb2803bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:26.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.803+0000 7fdb3ce95640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 0x7fdb2803bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0058e0 tx=0x7fdb2c005870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:26.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.808+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdb3000e0a0 con 0x7fdb3819c160 2024-09-16T01:02:26.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.981 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.982 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:26.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.985+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdb381a48a0 con 0x7fdb3819c160 2024-09-16T01:02:26.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.985+0000 7fdb35ffb640 1 -- 172.21.15.78:0/662465707 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fdb300112a0 con 0x7fdb3819c160 2024-09-16T01:02:26.986 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:26.986 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-09-16T01:02:26.986 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:02:26.987 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 msgr2=0x7fdb2803bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 0x7fdb2803bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0058e0 tx=0x7fdb2c005870 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 msgr2=0x7fdb3819c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000ba50 tx=0x7fdb3000bf10 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 shutdown_connections 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb28039730 0x7fdb2803bbf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3819c160 0x7fdb3819c610 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb38197890 0x7fdb3819bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 --2- 172.21.15.78:0/662465707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb38102730 0x7fdb38197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:26.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 >> 172.21.15.78:0/662465707 conn(0x7fdb38077310 msgr2=0x7fdb381035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:26.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 shutdown_connections 2024-09-16T01:02:26.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:26.989+0000 7fdb3f0f9640 1 -- 172.21.15.78:0/662465707 wait complete. 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.606 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-16T01:02:27.606 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:02:27.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T01:02:27.674 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:02:27.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:02:27.741 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T01:02:27.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T01:02:27.771 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T01:02:27.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:02:27.839 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T01:02:27.839 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-09-16T01:02:27.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:02:27.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:02:27.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:02:27.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T01:02:27.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:27.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:27.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:27.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: Reconfiguring daemon mon.a on smithi078 2024-09-16T01:02:27.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:27 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/662465707' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:27.867 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T01:02:27.867 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:02:27.934 INFO:tasks.cephadm:Adding mgr.x on smithi078 2024-09-16T01:02:27.935 INFO:tasks.cephadm:Adding mgr.y on smithi148 2024-09-16T01:02:27.935 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-09-16T01:02:27.935 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi078=x;smithi148=y;smithi193=z' 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: Reconfiguring daemon mon.a on smithi078 2024-09-16T01:02:27.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:27 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/662465707' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:28.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Reconfiguring mon.a (unknown last config time)... 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: Reconfiguring daemon mon.a on smithi078 2024-09-16T01:02:28.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:27 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/662465707' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:28.212 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:02:28.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3373505158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d81005e0 msgr2=0x7fc1d81009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:28.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3373505158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d81005e0 0x7fc1d81009e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009880 tx=0x7fc1cc02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3373505158 shutdown_connections 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3373505158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d81005e0 0x7fc1d81009e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3373505158 >> 172.21.15.193:0/3373505158 conn(0x7fc1d80fbdb0 msgr2=0x7fc1d80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3373505158 shutdown_connections 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.651+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3373505158 wait complete. 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 Processor -- start 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 -- start start 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1d81005e0 0x7fc1d8197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d8197580 0x7fc1d819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:28.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d81066d0 con 0x7fc1d8197a00 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8106550 con 0x7fc1d81005e0 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1df1cd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8106850 con 0x7fc1d8197580 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1dd76a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1dd76a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.193:44786/0 (socket says 172.21.15.193:44786) 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1d7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d8197580 0x7fc1d819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1dd76a640 1 -- 172.21.15.193:0/3276067768 learned_addr learned my addr 172.21.15.193:0/3276067768 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.652+0000 7fc1dcf69640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1d81005e0 0x7fc1d8197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 -- 172.21.15.193:0/3276067768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d8197580 msgr2=0x7fc1d819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d8197580 0x7fc1d819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 -- 172.21.15.193:0/3276067768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1d81005e0 msgr2=0x7fc1d8197040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1d81005e0 0x7fc1d8197040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 -- 172.21.15.193:0/3276067768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d819f440 con 0x7fc1d8197a00 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1dd76a640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000ba50 tx=0x7fc1d000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0019070 con 0x7fc1d8197a00 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d819f730 con 0x7fc1d8197a00 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1d000cbe0 con 0x7fc1d8197a00 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d00076a0 con 0x7fc1d8197a00 2024-09-16T01:02:28.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.653+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d819fcf0 con 0x7fc1d8197a00 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.654+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc1d001c070 con 0x7fc1d8197a00 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.654+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d8190860 con 0x7fc1d8197a00 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.654+0000 7fc1d5ffb640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 0x7fc1c803bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.655+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc1d004bf60 con 0x7fc1d8197a00 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.655+0000 7fc1dcf69640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 0x7fc1c803bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:28.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.655+0000 7fc1dcf69640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 0x7fc1c803bbf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009230 tx=0x7fc1cc0315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:28.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.659+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc1d000ce00 con 0x7fc1d8197a00 2024-09-16T01:02:28.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.838+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi148=y;smithi193=z", "target": ["mon-mgr", ""]}) -- 0x7fc1d80626c0 con 0x7fc1c8039730 2024-09-16T01:02:28.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.844+0000 7fc1d5ffb640 1 -- 172.21.15.193:0/3276067768 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc1d80626c0 con 0x7fc1c8039730 2024-09-16T01:02:28.845 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 msgr2=0x7fc1c803bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 0x7fc1c803bbf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009230 tx=0x7fc1cc0315d0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 msgr2=0x7fc1d819eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000ba50 tx=0x7fc1d000bf10 comp rx=0 tx=0).stop 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 shutdown_connections 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1c8039730 0x7fc1c803bbf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d8197a00 0x7fc1d819eb80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d8197580 0x7fc1d819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 --2- 172.21.15.193:0/3276067768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1d81005e0 0x7fc1d8197040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:28.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 >> 172.21.15.193:0/3276067768 conn(0x7fc1d80fbdb0 msgr2=0x7fc1d806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:28.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 shutdown_connections 2024-09-16T01:02:28.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:02:28.846+0000 7fc1df1cd640 1 -- 172.21.15.193:0/3276067768 wait complete. 2024-09-16T01:02:29.220 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:29.221 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:29.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:29.455 DEBUG:teuthology.orchestra.run.smithi148:mgr.y> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.y.service 2024-09-16T01:02:29.458 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.z.service 2024-09-16T01:02:29.460 INFO:tasks.cephadm:Deploying OSDs... 2024-09-16T01:02:29.460 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:02:29.460 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:02:29.477 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:02:29.478 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-16 01:01:48.652324148 +0000 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-16 00:57:37.872866417 +0000 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-16 00:57:37.872866417 +0000 2024-09-16T01:02:29.534 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-16 00:57:37.872866417 +0000 2024-09-16T01:02:29.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:02:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-16T01:02:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-16T01:02:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000223093 s, 2.3 MB/s 2024-09-16T01:02:29.601 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:02:29.658 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-16 01:01:48.690323565 +0000 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-16 00:57:38.201861384 +0000 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-16 00:57:38.201861384 +0000 2024-09-16T01:02:29.715 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-16 00:57:38.201861384 +0000 2024-09-16T01:02:29.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:02:29.778 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-16T01:02:29.779 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-16T01:02:29.779 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000215579 s, 2.4 MB/s 2024-09-16T01:02:29.780 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:02:29.836 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-16 01:01:48.725323028 +0000 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-16 00:57:38.531856336 +0000 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-16 00:57:38.531856336 +0000 2024-09-16T01:02:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-16 00:57:38.531856336 +0000 2024-09-16T01:02:29.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:02:29.955 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-16T01:02:29.955 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-16T01:02:29.955 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000302989 s, 1.7 MB/s 2024-09-16T01:02:29.957 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:02:30.013 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-09-16T01:02:30.069 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:02:30.069 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:30.069 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-16 01:01:48.758322521 +0000 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-16 00:57:38.874851089 +0000 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-16 00:57:38.874851089 +0000 2024-09-16T01:02:30.070 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-09-16 00:57:38.874851089 +0000 2024-09-16T01:02:30.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:02:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-16T01:02:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-16T01:02:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000196111 s, 2.6 MB/s 2024-09-16T01:02:30.134 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:02:30.191 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T01:02:30.191 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:02:30.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:02:30.208 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-09-16T01:02:30.231 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:29 smithi148 ceph-mon[35030]: Reconfiguring mon.b (monmap changed)... 2024-09-16T01:02:30.231 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:29 smithi148 ceph-mon[35030]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi148=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:30.231 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:29 smithi148 ceph-mon[35030]: Saving service mgr spec with placement smithi078=x;smithi148=y;smithi193=z;count:3 2024-09-16T01:02:30.231 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:29 smithi148 ceph-mon[35030]: Reconfiguring daemon mon.b on smithi148 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 01:02:18.950534164 +0000 2024-09-16T01:02:30.250 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 00:57:49.760073002 +0000 2024-09-16T01:02:30.251 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 00:57:49.760073002 +0000 2024-09-16T01:02:30.251 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 00:57:49.760073002 +0000 2024-09-16T01:02:30.251 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:02:30.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:29 smithi078 ceph-mon[30937]: Reconfiguring mon.b (monmap changed)... 2024-09-16T01:02:30.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:29 smithi078 ceph-mon[30937]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi148=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:30.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:29 smithi078 ceph-mon[30937]: Saving service mgr spec with placement smithi078=x;smithi148=y;smithi193=z;count:3 2024-09-16T01:02:30.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:29 smithi078 ceph-mon[30937]: Reconfiguring daemon mon.b on smithi148 2024-09-16T01:02:30.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:29 smithi193 ceph-mon[35701]: Reconfiguring mon.b (monmap changed)... 2024-09-16T01:02:30.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:29 smithi193 ceph-mon[35701]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi148=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:30.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:29 smithi193 ceph-mon[35701]: Saving service mgr spec with placement smithi078=x;smithi148=y;smithi193=z;count:3 2024-09-16T01:02:30.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:29 smithi193 ceph-mon[35701]: Reconfiguring daemon mon.b on smithi148 2024-09-16T01:02:30.324 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T01:02:30.324 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T01:02:30.324 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000346564 s, 1.5 MB/s 2024-09-16T01:02:30.327 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:02:30.386 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 01:02:18.951534138 +0000 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 00:57:50.119063849 +0000 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 00:57:50.119063849 +0000 2024-09-16T01:02:30.451 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 00:57:50.119063849 +0000 2024-09-16T01:02:30.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:02:30.520 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T01:02:30.520 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T01:02:30.520 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000207975 s, 2.5 MB/s 2024-09-16T01:02:30.521 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:02:30.581 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-09-16T01:02:30.639 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 01:02:18.951534138 +0000 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 00:57:50.429055947 +0000 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 00:57:50.429055947 +0000 2024-09-16T01:02:30.640 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 00:57:50.429055947 +0000 2024-09-16T01:02:30.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:02:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T01:02:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T01:02:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000214469 s, 2.4 MB/s 2024-09-16T01:02:30.707 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:02:30.766 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 01:02:18.952534113 +0000 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 00:57:50.749047790 +0000 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 00:57:50.749047790 +0000 2024-09-16T01:02:30.823 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 00:57:50.749047790 +0000 2024-09-16T01:02:30.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:02:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T01:02:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T01:02:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000177736 s, 2.9 MB/s 2024-09-16T01:02:30.889 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:02:30.946 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-09-16T01:02:30.946 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:02:30.963 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:02:30.963 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-09-16T01:02:31.018 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-09-16 01:02:09.857875322 +0000 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-09-16 00:57:37.251181188 +0000 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-09-16 00:57:37.251181188 +0000 2024-09-16T01:02:31.019 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-09-16 00:57:37.251181188 +0000 2024-09-16T01:02:31.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:02:31.083 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-09-16T01:02:31.084 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-09-16T01:02:31.084 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000195714 s, 2.6 MB/s 2024-09-16T01:02:31.085 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:02:31.141 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-09-16T01:02:31.197 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-09-16 01:02:09.857875322 +0000 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-09-16 00:57:37.590175718 +0000 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-09-16 00:57:37.590175718 +0000 2024-09-16T01:02:31.198 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-09-16 00:57:37.590175718 +0000 2024-09-16T01:02:31.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:02:31.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:31.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:31.260 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-09-16T01:02:31.261 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-09-16T01:02:31.261 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000261358 s, 2.0 MB/s 2024-09-16T01:02:31.262 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:31.319 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-09-16T01:02:31.375 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-09-16 01:02:09.857875322 +0000 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-09-16 00:57:37.896170781 +0000 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-09-16 00:57:37.896170781 +0000 2024-09-16T01:02:31.376 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-09-16 00:57:37.896170781 +0000 2024-09-16T01:02:31.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-16T01:02:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:31.439 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-09-16T01:02:31.439 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-09-16T01:02:31.439 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000195835 s, 2.6 MB/s 2024-09-16T01:02:31.440 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:02:31.497 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-09-16T01:02:31.553 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-09-16 01:02:09.858875306 +0000 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-09-16 00:57:38.225165473 +0000 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-09-16 00:57:38.225165473 +0000 2024-09-16T01:02:31.554 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-09-16 00:57:38.225165473 +0000 2024-09-16T01:02:31.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:02:31.617 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-09-16T01:02:31.618 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-09-16T01:02:31.618 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000227359 s, 2.3 MB/s 2024-09-16T01:02:31.619 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:02:31.674 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2024-09-16T01:02:31.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T01:02:31.918 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:02:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: Reconfiguring mon.c (monmap changed)... 2024-09-16T01:02:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: Reconfiguring daemon mon.c on smithi193 2024-09-16T01:02:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:32.284 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: Reconfiguring mon.c (monmap changed)... 2024-09-16T01:02:32.284 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: Reconfiguring daemon mon.c on smithi193 2024-09-16T01:02:32.284 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.284 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:32.285 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:32.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: Reconfiguring mon.c (monmap changed)... 2024-09-16T01:02:32.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: Reconfiguring daemon mon.c on smithi193 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:32.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:33.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:32 smithi078 ceph-mon[30937]: Deploying daemon mgr.z on smithi193 2024-09-16T01:02:33.169 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:32 smithi078 ceph-mon[30937]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:32 smithi193 ceph-mon[35701]: Deploying daemon mgr.z on smithi193 2024-09-16T01:02:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:32 smithi193 ceph-mon[35701]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:33.296 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 systemd[1]: Starting Ceph mgr.z for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:02:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:32 smithi148 ceph-mon[35030]: Deploying daemon mgr.z on smithi193 2024-09-16T01:02:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:32 smithi148 ceph-mon[35030]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 podman[37492]: 2024-09-16 01:02:33.389725795 +0000 UTC m=+0.105904456 container create 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 podman[37492]: 2024-09-16 01:02:33.315401358 +0000 UTC m=+0.031579977 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 podman[37492]: 2024-09-16 01:02:33.583579591 +0000 UTC m=+0.299758214 container init 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True) 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 podman[37492]: 2024-09-16 01:02:33.587075249 +0000 UTC m=+0.303253871 container start 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 ceph-mgr[37511]: -- 172.21.15.193:0/40793560 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559abb7770e0 con 0x559abb754800 2024-09-16T01:02:33.643 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 bash[37492]: 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 2024-09-16T01:02:33.917 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 systemd[1]: Started Ceph mgr.z for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:02:33.918 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:33.713+0000 7f51a7509040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:02:33.918 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:33.770+0000 7f51a7509040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:02:33.918 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:33.916+0000 7f51a7509040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:02:34.261 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:02:34.266 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:33.982+0000 7f51a7509040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:02:34.266 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:34.128+0000 7f51a7509040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:02:34.266 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:34.265+0000 7f51a7509040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:02:34.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-09-16T01:02:34.525 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:02:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:34.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:34.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:34.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:34.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: Deploying daemon mgr.y on smithi148 2024-09-16T01:02:34.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:34 smithi148 ceph-mon[35030]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:34.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:34.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:34.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:34.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:34.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: Deploying daemon mgr.y on smithi148 2024-09-16T01:02:34.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:34 smithi078 ceph-mon[30937]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 -- 172.21.15.78:0/593439613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870102690 msgr2=0x7ff870108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 --2- 172.21.15.78:0/593439613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870102690 0x7ff870108f20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff868009ca0 tx=0x7ff86802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 -- 172.21.15.78:0/593439613 shutdown_connections 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 --2- 172.21.15.78:0/593439613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870102690 0x7ff870108f20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 --2- 172.21.15.78:0/593439613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 0x7ff870102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 --2- 172.21.15.78:0/593439613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870100ad0 0x7ff870100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 -- 172.21.15.78:0/593439613 >> 172.21.15.78:0/593439613 conn(0x7ff8700fc280 msgr2=0x7ff8700fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 -- 172.21.15.78:0/593439613 shutdown_connections 2024-09-16T01:02:34.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.959+0000 7ff8766ef640 1 -- 172.21.15.78:0/593439613 wait complete. 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 Processor -- start 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 -- start start 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 0x7ff87019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870102690 0x7ff87019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010c750 con 0x7ff870100ad0 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010c5d0 con 0x7ff870102690 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff8766ef640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010c8d0 con 0x7ff870101cd0 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff86ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff86ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40852/0 (socket says 172.21.15.78:40852) 2024-09-16T01:02:34.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff86ffff640 1 -- 172.21.15.78:0/3018688384 learned_addr learned my addr 172.21.15.78:0/3018688384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff867fff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 0x7ff87019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.960+0000 7ff874c8c640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870102690 0x7ff87019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 -- 172.21.15.78:0/3018688384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 msgr2=0x7ff87019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 0x7ff87019cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 -- 172.21.15.78:0/3018688384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870102690 msgr2=0x7ff87019f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870102690 0x7ff87019f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 -- 172.21.15.78:0/3018688384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87019fb00 con 0x7ff870100ad0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86ffff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff86000e9c0 tx=0x7ff86000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86000cd80 con 0x7ff870100ad0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8600035f0 con 0x7ff870100ad0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87019fd90 con 0x7ff870100ad0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff860010690 con 0x7ff870100ad0 2024-09-16T01:02:34.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.961+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8701a0250 con 0x7ff870100ad0 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7ff860010830 con 0x7ff870100ad0 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff870107e50 con 0x7ff870100ad0 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff86dffb640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 0x7ff85c03b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff86004ca50 con 0x7ff870100ad0 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff867fff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 0x7ff85c03b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:02:34.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.962+0000 7ff867fff640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 0x7ff85c03b990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff87006b050 tx=0x7ff858005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:02:34.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:34.967+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff860003b30 con 0x7ff870100ad0 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: Deploying daemon mgr.y on smithi148 2024-09-16T01:02:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-mon[35701]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:35.047 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:34.581+0000 7f51a7509040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:02:35.047 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:34.641+0000 7f51a7509040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:02:35.047 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:34 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:34.825+0000 7f51a7509040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:02:35.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:35.141+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff870062630 con 0x7ff85c0394d0 2024-09-16T01:02:35.730 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:35 smithi148 systemd[1]: Starting Ceph mgr.y for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:02:35.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:02:35.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:02:35.854 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:35 smithi148 podman[36380]: 2024-09-16 01:02:35.78225604 +0000 UTC m=+0.112026069 container create 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:35 smithi148 podman[36380]: 2024-09-16 01:02:35.686098948 +0000 UTC m=+0.015868986 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:02:35.985 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:35 smithi148 podman[36380]: 2024-09-16 01:02:35.984373827 +0000 UTC m=+0.314143856 container init 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-16T01:02:36.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:02:36.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:02:36.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:36.047 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:35.705+0000 7f51a7509040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:02:36.047 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:35.773+0000 7f51a7509040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:35 smithi148 podman[36380]: 2024-09-16 01:02:35.987104731 +0000 UTC m=+0.316874768 container start 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mgr[36400]: -- 172.21.15.148:0/1399665129 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b48a5870e0 con 0x55b48a565400 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 bash[36380]: 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 systemd[1]: Started Ceph mgr.y for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.110+0000 7f750e59d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.181+0000 7f750e59d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:02:36.341 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.340+0000 7f750e59d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:02:36.635 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.409+0000 7f750e59d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:02:36.635 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.568+0000 7f750e59d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:02:36.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:36.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:36.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:36.506+0000 7f51a7509040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:36.637+0000 7f51a7509040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:02:36.791 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:36.789+0000 7f51a7509040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:02:36.986 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:36.736+0000 7f750e59d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:02:36.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:36.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:36.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.986 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.987 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.987 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:36.987 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:37.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:37.046 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:36.851+0000 7f51a7509040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:02:37.046 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.009+0000 7f51a7509040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:02:37.313 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.080+0000 7f51a7509040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:02:37.314 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.140+0000 7f51a7509040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:02:37.314 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.312+0000 7f51a7509040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:02:37.323 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:37.063+0000 7f750e59d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:02:37.323 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:37.125+0000 7f750e59d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:02:37.323 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:37.322+0000 7f750e59d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:02:37.591 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.379+0000 7f51a7509040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:02:37.591 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.447+0000 7f51a7509040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:02:37.591 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.531+0000 7f51a7509040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:02:37.591 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.589+0000 7f51a7509040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:02:38.045 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:37.649+0000 7f51a7509040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:02:38.635 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:38.215+0000 7f750e59d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:02:38.635 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:38.282+0000 7f750e59d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:02:38.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]: dispatch 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]': finished 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:38.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: Reconfiguring daemon mgr.x on smithi078 2024-09-16T01:02:38.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:38.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/194877490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:02:38.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]: dispatch 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]': finished 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:38.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: Reconfiguring daemon mgr.x on smithi078 2024-09-16T01:02:38.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:38.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/194877490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:02:38.438+0000 7f51a7509040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1798893848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ded124f4-818a-4255-a9cb-dcaa0fc71556"}]': finished 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: osdmap e5: 1 total, 0 up, 1 in 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: Reconfiguring mgr.x (unknown last config time)... 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: Reconfiguring daemon mgr.x on smithi078 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:38.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/194877490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:02:39.255 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:38.997+0000 7f750e59d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:02:39.256 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.111+0000 7f750e59d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:02:39.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:39.429+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7ff86001dd20 con 0x7ff870100ad0 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-mon[35030]: Standby manager daemon z started 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.255+0000 7f750e59d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:02:39.534 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.316+0000 7f750e59d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:02:39.535 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.465+0000 7f750e59d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:02:39.535 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.533+0000 7f750e59d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:02:39.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:39 smithi078 ceph-mon[30937]: Standby manager daemon z started 2024-09-16T01:02:39.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:39 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:02:39.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:39 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:39.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:39 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:02:39.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:39 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:39.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:39 smithi193 ceph-mon[35701]: Standby manager daemon z started 2024-09-16T01:02:39.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:39 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:02:39.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:39 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:39.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:39 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:02:39.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:39 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/481923750' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:39.814 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.592+0000 7f750e59d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:02:39.814 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.748+0000 7f750e59d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:02:39.814 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.813+0000 7f750e59d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:02:40.078 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.879+0000 7f750e59d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:02:40.078 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:39.961+0000 7f750e59d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:02:40.078 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:40.019+0000 7f750e59d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:02:40.078 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:40.077+0000 7f750e59d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:02:40.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:40 smithi078 ceph-mon[30937]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-16T01:02:40.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:02:40.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:40 smithi078 ceph-mon[30937]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:40.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:40 smithi193 ceph-mon[35701]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-16T01:02:40.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:02:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:40 smithi193 ceph-mon[35701]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:40.864 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-mon[35030]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-16T01:02:40.864 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:02:40.864 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-mon[35030]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:41.135 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:02:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:02:40.862+0000 7f750e59d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:02:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:41.439+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7ff860018ce0 con 0x7ff870100ad0 2024-09-16T01:02:41.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:41 smithi078 ceph-mon[30937]: Standby manager daemon y started 2024-09-16T01:02:41.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:41 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:02:41.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:41 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:41.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:41 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:02:41.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:41 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:41.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:41 smithi193 ceph-mon[35701]: Standby manager daemon y started 2024-09-16T01:02:41.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:41 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:02:41.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:41 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:41.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:41 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:02:41.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:41 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:41.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:41 smithi148 ceph-mon[35030]: Standby manager daemon y started 2024-09-16T01:02:41.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:41 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:02:41.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:41 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:02:41.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:41 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:02:41.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:41 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/2688621827' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:02:42.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:42 smithi078 ceph-mon[30937]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-16T01:02:42.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:42 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:02:42.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:42 smithi078 ceph-mon[30937]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:42 smithi193 ceph-mon[35701]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-16T01:02:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:42 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:02:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:42 smithi193 ceph-mon[35701]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:42 smithi148 ceph-mon[35030]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-16T01:02:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:42 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:02:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:42 smithi148 ceph-mon[35030]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:47.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:46 smithi078 ceph-mon[30937]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:47.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T01:02:47.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:46 smithi193 ceph-mon[35701]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T01:02:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:46 smithi148 ceph-mon[35030]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-16T01:02:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:48.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:47 smithi078 ceph-mon[30937]: Deploying daemon osd.0 on smithi078 2024-09-16T01:02:48.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:47 smithi193 ceph-mon[35701]: Deploying daemon osd.0 on smithi078 2024-09-16T01:02:48.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:47 smithi148 ceph-mon[35030]: Deploying daemon osd.0 on smithi078 2024-09-16T01:02:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:48 smithi078 ceph-mon[30937]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:49.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:48 smithi193 ceph-mon[35701]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:48 smithi148 ceph-mon[35030]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:51.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:50 smithi078 ceph-mon[30937]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:51.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:50 smithi193 ceph-mon[35701]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:50 smithi148 ceph-mon[35030]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:52 smithi078 ceph-mon[30937]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:52 smithi193 ceph-mon[35701]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:52 smithi148 ceph-mon[35030]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:54 smithi078 ceph-mon[30937]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:55.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:54 smithi193 ceph-mon[35701]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:54 smithi148 ceph-mon[35030]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:56 smithi078 ceph-mon[30937]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:56 smithi193 ceph-mon[35701]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:56.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:02:56.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:02:56.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:56.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:56 smithi148 ceph-mon[35030]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:57.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.082+0000 7ff86dffb640 1 -- 172.21.15.78:0/3018688384 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff870062630 con 0x7ff85c0394d0 2024-09-16T01:02:57.084 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 msgr2=0x7ff85c03b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 0x7ff85c03b990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff87006b050 tx=0x7ff858005f90 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 msgr2=0x7ff87006aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff86000e9c0 tx=0x7ff86000ee80 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 shutdown_connections 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff85c0394d0 0x7ff85c03b990 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff870102690 0x7ff87019f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff870101cd0 0x7ff87019cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 --2- 172.21.15.78:0/3018688384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff870100ad0 0x7ff87006aad0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 >> 172.21.15.78:0/3018688384 conn(0x7ff8700fc280 msgr2=0x7ff870105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 shutdown_connections 2024-09-16T01:02:57.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:02:57.085+0000 7ff8766ef640 1 -- 172.21.15.78:0/3018688384 wait complete. 2024-09-16T01:02:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:02:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:02:57.765 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.0.service 2024-09-16T01:02:57.767 INFO:tasks.cephadm:Deploying osd.1 on smithi148 with /dev/vg_nvme/lv_4... 2024-09-16T01:02:57.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T01:02:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:02:58.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:58 smithi148 ceph-mon[35030]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:58 smithi078 ceph-mon[30937]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:58.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:58 smithi193 ceph-mon[35701]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:02:59.235 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:02:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:02:58.909+0000 7f6473d49640 -1 osd.0 0 log_to_monitors true 2024-09-16T01:02:59.357 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:02:59 smithi148 ceph-mon[35030]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T01:02:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:02:59 smithi078 ceph-mon[30937]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T01:02:59.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:02:59 smithi193 ceph-mon[35701]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-16T01:03:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:00 smithi148 ceph-mon[35030]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T01:03:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:00 smithi148 ceph-mon[35030]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T01:03:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:00 smithi148 ceph-mon[35030]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:03:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:00.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:00 smithi148 ceph-mon[35030]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:00.436 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T01:03:00.465 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_4 2024-09-16T01:03:00.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-mon[30937]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T01:03:00.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-mon[30937]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T01:03:00.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-mon[30937]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:03:00.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:00.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-mon[30937]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:00.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:00 smithi193 ceph-mon[35701]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-16T01:03:00.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:00 smithi193 ceph-mon[35701]: osdmap e6: 1 total, 0 up, 1 in 2024-09-16T01:03:00.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:00 smithi193 ceph-mon[35701]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:03:00.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:00.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:00 smithi193 ceph-mon[35701]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:00.727 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:03:00.881 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:03:00.642+0000 7f646fccc640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T01:03:00.882 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:03:00.645+0000 7f6467aec640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:03:01.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.161+0000 7f3837c63640 1 -- 172.21.15.148:0/1226038708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830100af0 msgr2=0x7f3830100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.161+0000 7f3837c63640 1 --2- 172.21.15.148:0/1226038708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830100af0 0x7f3830100ef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3824009900 tx=0x7f382402a410 comp rx=0 tx=0).stop 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 -- 172.21.15.148:0/1226038708 shutdown_connections 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 --2- 172.21.15.148:0/1226038708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38301026b0 0x7f3830108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 --2- 172.21.15.148:0/1226038708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3830101cf0 0x7f3830102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 --2- 172.21.15.148:0/1226038708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830100af0 0x7f3830100ef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 -- 172.21.15.148:0/1226038708 >> 172.21.15.148:0/1226038708 conn(0x7f38300fc280 msgr2=0x7f38300fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 -- 172.21.15.148:0/1226038708 shutdown_connections 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 -- 172.21.15.148:0/1226038708 wait complete. 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 Processor -- start 2024-09-16T01:03:01.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.162+0000 7f3837c63640 1 -- start start 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 0x7f3830104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 0x7f3830103610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f383010c770 con 0x7f3830100af0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f383010c5f0 con 0x7f3830101cf0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3837c63640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f383010c8f0 con 0x7f38301026b0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44730/0 (socket says 172.21.15.148:44730) 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 -- 172.21.15.148:0/341563888 learned_addr learned my addr 172.21.15.148:0/341563888 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38359ff640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 0x7f3830104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3836200640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 0x7f3830103610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 -- 172.21.15.148:0/341563888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 msgr2=0x7f3830103610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:01.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 0x7f3830103610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 -- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 msgr2=0x7f3830104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 0x7f3830104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 -- 172.21.15.148:0/341563888 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3830103ed0 con 0x7f3830101cf0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3836200640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 0x7f3830103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38359ff640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 0x7f3830104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f38351fe640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d770 tx=0x7f382c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c017070 con 0x7f3830101cf0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.163+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f382c005500 con 0x7f3830101cf0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.164+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38301a2080 con 0x7f3830101cf0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.164+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c004e00 con 0x7f3830101cf0 2024-09-16T01:03:01.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.164+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38301a2540 con 0x7f3830101cf0 2024-09-16T01:03:01.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.165+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3830100ef0 con 0x7f3830101cf0 2024-09-16T01:03:01.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.166+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f382c004fa0 con 0x7f3830101cf0 2024-09-16T01:03:01.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.167+0000 7f3822ffd640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 0x7f381c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:01.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.167+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7f382c0cdb90 con 0x7f3830101cf0 2024-09-16T01:03:01.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.167+0000 7f38359ff640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 0x7f381c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:01.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.168+0000 7f38359ff640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 0x7f381c0a5670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3824005ab0 tx=0x7f3824000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:01.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.170+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f382c09c440 con 0x7f3830101cf0 2024-09-16T01:03:01.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:01.353+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f383006aed0 con 0x7f381c0a31b0 2024-09-16T01:03:01.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:03:01.354 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi078 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 17088M 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:01.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.356 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:01 smithi148 ceph-mon[35030]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' 2024-09-16T01:03:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:03:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T01:03:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi078 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 17088M 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:01 smithi078 ceph-mon[30937]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' 2024-09-16T01:03:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:03:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: osdmap e7: 1 total, 0 up, 1 in 2024-09-16T01:03:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi078 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 17088M 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:01.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:01.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:01.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:01 smithi193 ceph-mon[35701]: from='osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933]' entity='osd.0' 2024-09-16T01:03:02.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] boot 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:02 smithi148 ceph-mon[35030]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:02.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] boot 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:02 smithi078 ceph-mon[30937]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: osd.0 [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] boot 2024-09-16T01:03:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: osdmap e8: 1 total, 1 up, 1 in 2024-09-16T01:03:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:03:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:02 smithi193 ceph-mon[35701]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-16T01:03:03.905 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T01:03:03.906 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/1428663058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:03.906 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:03.906 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]': finished 2024-09-16T01:03:03.906 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T01:03:03.906 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:03 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/1428663058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]': finished 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T01:03:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:03 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: osdmap e9: 1 total, 1 up, 1 in 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/1428663058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]: dispatch 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82da4924-eaf5-44df-adce-7f605462c130"}]': finished 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: osdmap e10: 2 total, 1 up, 2 in 2024-09-16T01:03:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:03 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:04 smithi078 ceph-mon[30937]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/2939583412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:05.045 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:04 smithi193 ceph-mon[35701]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:04 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/2939583412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:04 smithi148 ceph-mon[35030]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:04 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/2939583412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:06 smithi078 ceph-mon[30937]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:06 smithi193 ceph-mon[35701]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:07.388 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:06 smithi148 ceph-mon[35030]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:09.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:08 smithi148 ceph-mon[35030]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:08 smithi078 ceph-mon[30937]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:09.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:08 smithi193 ceph-mon[35701]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:10 smithi148 ceph-mon[35030]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:10 smithi078 ceph-mon[30937]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:11.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:10 smithi193 ceph-mon[35701]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:12.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T01:03:12.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:12.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T01:03:12.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:12.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-16T01:03:12.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:13.113 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:12 smithi148 ceph-mon[35030]: Deploying daemon osd.1 on smithi148 2024-09-16T01:03:13.113 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:12 smithi148 ceph-mon[35030]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:12 smithi078 ceph-mon[30937]: Deploying daemon osd.1 on smithi148 2024-09-16T01:03:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:12 smithi078 ceph-mon[30937]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:13.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:12 smithi193 ceph-mon[35701]: Deploying daemon osd.1 on smithi148 2024-09-16T01:03:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:12 smithi193 ceph-mon[35701]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:14 smithi078 ceph-mon[30937]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:15.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:14 smithi193 ceph-mon[35701]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:14 smithi148 ceph-mon[35030]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:17.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:17 smithi148 ceph-mon[35030]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:16 smithi078 ceph-mon[30937]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:16 smithi193 ceph-mon[35701]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:18.276 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:18.276 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:18.276 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:18.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:18.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:18.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:18.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:18.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:18.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:19.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:19 smithi148 ceph-mon[35030]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:19 smithi078 ceph-mon[30937]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:19 smithi193 ceph-mon[35701]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:21.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:21.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:21.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:20 smithi148 ceph-mon[35030]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:21.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:20 smithi078 ceph-mon[30937]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:21.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:20 smithi193 ceph-mon[35701]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:21.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.588+0000 7f3822ffd640 1 -- 172.21.15.148:0/341563888 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f383006aed0 con 0x7f381c0a31b0 2024-09-16T01:03:21.590 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 1 on host 'smithi148' 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 msgr2=0x7f381c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 0x7f381c0a5670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3824005ab0 tx=0x7f3824000bc0 comp rx=0 tx=0).stop 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 msgr2=0x7f38301030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d770 tx=0x7f382c00dc30 comp rx=0 tx=0).stop 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 shutdown_connections 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f381c0a31b0 0x7f381c0a5670 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38301026b0 0x7f3830103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3830101cf0 0x7f38301030d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:21.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 --2- 172.21.15.148:0/341563888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3830100af0 0x7f3830104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:21.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 >> 172.21.15.148:0/341563888 conn(0x7f38300fc280 msgr2=0x7f383006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:03:21.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 shutdown_connections 2024-09-16T01:03:21.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:03:21.590+0000 7f3837c63640 1 -- 172.21.15.148:0/341563888 wait complete. 2024-09-16T01:03:22.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:22.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.171 DEBUG:teuthology.orchestra.run.smithi148:osd.1> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.1.service 2024-09-16T01:03:22.173 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-09-16T01:03:22.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-16T01:03:22.202 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:22.202 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.202 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:22.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:22.424 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:03:23.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:22 smithi148 ceph-mon[35030]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:22 smithi078 ceph-mon[30937]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:22 smithi193 ceph-mon[35701]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:23.873 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:23 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:03:23.557+0000 7f815cb1f640 -1 osd.1 0 log_to_monitors true 2024-09-16T01:03:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:23 smithi193 ceph-mon[35701]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:23 smithi193 ceph-mon[35701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:23 smithi148 ceph-mon[35030]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:23 smithi148 ceph-mon[35030]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:23 smithi078 ceph-mon[30937]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:23 smithi078 ceph-mon[30937]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-16T01:03:24.732 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:03:24.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-09-16T01:03:25.048 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.169 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:24 smithi193 ceph-mon[35701]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:24 smithi078 ceph-mon[30937]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:25.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-16T01:03:25.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: osdmap e11: 2 total, 1 up, 2 in 2024-09-16T01:03:25.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:25.284 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: from='osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:03:25.285 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:24 smithi148 ceph-mon[35030]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:25.483 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- 172.21.15.193:0/1658864612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40102690 msgr2=0x7f1d40104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 --2- 172.21.15.193:0/1658864612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40102690 0x7f1d40104a80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009900 tx=0x7f1d3c02a410 comp rx=0 tx=0).stop 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- 172.21.15.193:0/1658864612 shutdown_connections 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 --2- 172.21.15.193:0/1658864612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 0x7f1d40109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 --2- 172.21.15.193:0/1658864612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40104fc0 0x7f1d401073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 --2- 172.21.15.193:0/1658864612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40102690 0x7f1d40104a80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- 172.21.15.193:0/1658864612 >> 172.21.15.193:0/1658864612 conn(0x7f1d400fc290 msgr2=0x7f1d400fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- 172.21.15.193:0/1658864612 shutdown_connections 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- 172.21.15.193:0/1658864612 wait complete. 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 Processor -- start 2024-09-16T01:03:25.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.481+0000 7f1d48469640 1 -- start start 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40102690 0x7f1d40197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 0x7f1d4019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4010d910 con 0x7f1d40102690 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4010d790 con 0x7f1d401078f0 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4010da90 con 0x7f1d40104fc0 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:41120/0 (socket says 172.21.15.193:41120) 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 -- 172.21.15.193:0/2779099296 learned_addr learned my addr 172.21.15.193:0/2779099296 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d46a06640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 0x7f1d4019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d46205640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40102690 0x7f1d40197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:25.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 -- 172.21.15.193:0/2779099296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 msgr2=0x7f1d4019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 0x7f1d4019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 -- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40102690 msgr2=0x7f1d40197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40102690 0x7f1d40197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 -- 172.21.15.193:0/2779099296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4019fba0 con 0x7f1d40104fc0 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d45a04640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000d770 tx=0x7f1d3000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30017070 con 0x7f1d40104fc0 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d30009c30 con 0x7f1d40104fc0 2024-09-16T01:03:25.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.482+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d4019fe90 con 0x7f1d40104fc0 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.483+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3000f620 con 0x7f1d40104fc0 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.483+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d401a0390 con 0x7f1d40104fc0 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.484+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d401053a0 con 0x7f1d40104fc0 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.485+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f1d3001a070 con 0x7f1d40104fc0 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.486+0000 7f1d377fe640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 0x7f1d2c0a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:25.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.486+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2029+0+0 (secure 0 0 0) 0x7f1d300cd310 con 0x7f1d40104fc0 2024-09-16T01:03:25.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.486+0000 7f1d46205640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 0x7f1d2c0a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:25.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.486+0000 7f1d46205640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 0x7f1d2c0a5550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c006010 tx=0x7f1d3c031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:25.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.490+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1d3009b9a0 con 0x7f1d40104fc0 2024-09-16T01:03:25.635 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:03:25.282+0000 7f8158aa2640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T01:03:25.635 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:03:25.289+0000 7f81508c2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:03:25.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:25.664+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1d40062630 con 0x7f1d2c0a3090 2024-09-16T01:03:26.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi148 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='osd.1 ' entity='osd.1' 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:26.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi148 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.156 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='osd.1 ' entity='osd.1' 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:26.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:03:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: osdmap e12: 2 total, 1 up, 2 in 2024-09-16T01:03:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi148 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='osd.1 ' entity='osd.1' 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:26.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:27.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:03:27.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:03:27.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:27.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:27.192 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:27.193 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] boot 2024-09-16T01:03:27.193 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T01:03:27.193 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:03:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:03:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] boot 2024-09-16T01:03:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T01:03:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:03:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:03:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:27.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:27.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-16T01:03:27.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: osd.1 [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] boot 2024-09-16T01:03:27.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: osdmap e13: 2 total, 2 up, 2 in 2024-09-16T01:03:27.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/2335465632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]': finished 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:28.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:28 smithi193 ceph-mon[35701]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:28.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/2335465632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]': finished 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:28.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:28 smithi148 ceph-mon[35030]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:28.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: osdmap e14: 2 total, 2 up, 2 in 2024-09-16T01:03:28.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/2335465632' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]: dispatch 2024-09-16T01:03:28.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51654cac-6b84-4b81-9861-6218a4f82eab"}]': finished 2024-09-16T01:03:28.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: osdmap e15: 3 total, 2 up, 3 in 2024-09-16T01:03:28.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:28.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:28 smithi078 ceph-mon[30937]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:29 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/804328603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:29.615 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:29 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/804328603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:29.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:29 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/804328603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:30.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:30 smithi148 ceph-mon[35030]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:30.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:30 smithi078 ceph-mon[30937]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:30.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:30 smithi193 ceph-mon[35701]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:32 smithi078 ceph-mon[30937]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:33.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:32 smithi193 ceph-mon[35701]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:32 smithi148 ceph-mon[35030]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:34 smithi078 ceph-mon[30937]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:35.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:34 smithi193 ceph-mon[35701]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:34 smithi148 ceph-mon[35030]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:36 smithi193 ceph-mon[35701]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T01:03:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:36 smithi078 ceph-mon[30937]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T01:03:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:36 smithi148 ceph-mon[35030]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-16T01:03:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:37 smithi078 ceph-mon[30937]: Deploying daemon osd.2 on smithi193 2024-09-16T01:03:38.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:37 smithi193 ceph-mon[35701]: Deploying daemon osd.2 on smithi193 2024-09-16T01:03:38.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:37 smithi148 ceph-mon[35030]: Deploying daemon osd.2 on smithi193 2024-09-16T01:03:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:38 smithi193 ceph-mon[35701]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:39.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:38 smithi078 ceph-mon[30937]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:38 smithi148 ceph-mon[35030]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:41.045 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:41 smithi193 ceph-mon[35701]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:41.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:41 smithi078 ceph-mon[30937]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:41 smithi148 ceph-mon[35030]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:43 smithi078 ceph-mon[30937]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:43.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:43 smithi193 ceph-mon[35701]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:43 smithi148 ceph-mon[35030]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:45.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:45 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:45 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:45 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:45.576 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.575+0000 7f1d377fe640 1 -- 172.21.15.193:0/2779099296 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1d40062630 con 0x7f1d2c0a3090 2024-09-16T01:03:45.577 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 msgr2=0x7f1d2c0a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 0x7f1d2c0a5550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c006010 tx=0x7f1d3c031510 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 msgr2=0x7f1d401984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000d770 tx=0x7f1d3000dc30 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 shutdown_connections 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1d2c0a3090 0x7f1d2c0a5550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d401078f0 0x7f1d4019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d40104fc0 0x7f1d401984b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 --2- 172.21.15.193:0/2779099296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d40102690 0x7f1d40197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 >> 172.21.15.193:0/2779099296 conn(0x7f1d400fc290 msgr2=0x7f1d40103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:03:45.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 shutdown_connections 2024-09-16T01:03:45.580 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:03:45.578+0000 7f1d48469640 1 -- 172.21.15.193:0/2779099296 wait complete. 2024-09-16T01:03:46.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:46.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.194 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.2.service 2024-09-16T01:03:46.196 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2024-09-16T01:03:46.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T01:03:46.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:46.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:03:46.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:46.439 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:03:47.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:47 smithi078 ceph-mon[30937]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:47.286 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:47 smithi148 ceph-mon[35030]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:47 smithi193 ceph-mon[35701]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:48.013 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:47 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:03:47.791+0000 7f470ab30640 -1 osd.2 0 log_to_monitors true 2024-09-16T01:03:48.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:48 smithi078 ceph-mon[30937]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:48 smithi078 ceph-mon[30937]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:48 smithi193 ceph-mon[35701]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:48 smithi193 ceph-mon[35701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:48 smithi148 ceph-mon[35030]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:48 smithi148 ceph-mon[35030]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-16T01:03:48.954 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:03:48.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-09-16T01:03:49.230 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:03:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.295 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:03:49.023+0000 7f4706ab3640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T01:03:49.296 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:03:49.028+0000 7f47028db640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:49.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:49.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:49.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-16T01:03:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: osdmap e16: 3 total, 2 up, 3 in 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:03:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/1409100894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a4102e00 msgr2=0x7fd7a4103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/1409100894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a4102e00 0x7fd7a4103280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd798009900 tx=0x7fd79802a410 comp rx=0 tx=0).stop 2024-09-16T01:03:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/1409100894 shutdown_connections 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/1409100894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a41037c0 0x7fd7a410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/1409100894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a4102e00 0x7fd7a4103280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/1409100894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4101c00 0x7fd7a4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/1409100894 >> 172.21.15.78:0/1409100894 conn(0x7fd7a40fd390 msgr2=0x7fd7a40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/1409100894 shutdown_connections 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.720+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/1409100894 wait complete. 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 Processor -- start 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 -- start start 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a4101c00 0x7fd7a4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 0x7fd7a41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410d980 con 0x7fd7a41037c0 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410d800 con 0x7fd7a4101c00 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7aa0ba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a410db00 con 0x7fd7a4102e00 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48944/0 (socket says 172.21.15.78:48944) 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a3fff640 1 -- 172.21.15.78:0/156028804 learned_addr learned my addr 172.21.15.78:0/156028804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a37fe640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a4101c00 0x7fd7a4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a2ffd640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 0x7fd7a41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.721+0000 7fd7a3fff640 1 -- 172.21.15.78:0/156028804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 msgr2=0x7fd7a41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a3fff640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 0x7fd7a41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a3fff640 1 -- 172.21.15.78:0/156028804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a4101c00 msgr2=0x7fd7a4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a3fff640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a4101c00 0x7fd7a4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a3fff640 1 -- 172.21.15.78:0/156028804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a419fcc0 con 0x7fd7a41037c0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a2ffd640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 0x7fd7a41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a3fff640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b970 tx=0x7fd79c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c019070 con 0x7fd7a41037c0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c014440 con 0x7fd7a41037c0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a419ffb0 con 0x7fd7a41037c0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0075d0 con 0x7fd7a41037c0 2024-09-16T01:03:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.722+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a04e0 con 0x7fd7a41037c0 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.723+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a4109070 con 0x7fd7a41037c0 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.724+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7fd79c01c070 con 0x7fd7a41037c0 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.725+0000 7fd7a0ff9640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 0x7fd78c0a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.725+0000 7fd7a37fe640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 0x7fd78c0a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.725+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2571+0+0 (secure 0 0 0) 0x7fd79c0cdb80 con 0x7fd7a41037c0 2024-09-16T01:03:49.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.725+0000 7fd7a37fe640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 0x7fd78c0a5720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4102c60 tx=0x7fd794009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:03:49.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.728+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd79c09bff0 con 0x7fd7a41037c0 2024-09-16T01:03:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:49.900+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd7a4062630 con 0x7fd78c0a3260 2024-09-16T01:03:50.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi193 2024-09-16T01:03:50.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 17088M 2024-09-16T01:03:50.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:03:50.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T01:03:50.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:50.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:50.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi193 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 17088M 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:50.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:50.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi193 2024-09-16T01:03:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 17088M 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: osdmap e17: 3 total, 2 up, 3 in 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:03:50.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] boot 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T01:03:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:51 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:51.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:03:51.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:03:51.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:51.332 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:51.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] boot 2024-09-16T01:03:51.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T01:03:51.333 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: osd.2 [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] boot 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: osdmap e18: 3 total, 3 up, 3 in 2024-09-16T01:03:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:51 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:03:52.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:52 smithi078 ceph-mon[30937]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T01:03:52.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:52 smithi193 ceph-mon[35701]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T01:03:52.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:52 smithi148 ceph-mon[35030]: osdmap e19: 3 total, 3 up, 3 in 2024-09-16T01:03:53.005 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:52 smithi078 sudo[50076]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:53.005 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:52 smithi078 sudo[50076]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:53.005 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:52 smithi078 sudo[50076]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:53.005 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:52 smithi078 sudo[50076]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:53.278 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:53 smithi078 sudo[50082]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:53.278 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:53 smithi078 sudo[50082]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:53.279 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:53 smithi078 sudo[50082]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:53.279 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:03:53 smithi078 sudo[50082]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: pgmap v76: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]: dispatch 2024-09-16T01:03:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]': finished 2024-09-16T01:03:53.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T01:03:53.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:53.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1234432024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:53.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: pgmap v76: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]: dispatch 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]': finished 2024-09-16T01:03:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T01:03:53.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:53.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1234432024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:53.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:53.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: pgmap v76: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:53.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.568 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: osdmap e20: 3 total, 3 up, 3 in 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]: dispatch 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1414746490' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3aa4ba88-9fb8-4dfc-bc12-f313c953351a"}]': finished 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: osdmap e21: 4 total, 3 up, 4 in 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1234432024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:03:53.569 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:53.569 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43173]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:53.569 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43173]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:53.569 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43173]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:53.570 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43173]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:53.885 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43176]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:53.885 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43176]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:53.885 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43176]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:53.885 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:03:53 smithi148 sudo[43176]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:54.093 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:53 smithi193 sudo[44441]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:54.093 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:53 smithi193 sudo[44441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:54.093 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:53 smithi193 sudo[44441]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:54.093 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:53 smithi193 sudo[44441]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:54.546 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44444]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:54.546 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:54.546 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44444]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:54.546 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44444]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:54.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:54.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:54.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50316]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:54.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50316]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:03:54.645+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd79c097f60 con 0x7fd7a41037c0 2024-09-16T01:03:54.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T01:03:54.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44447]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44447]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44447]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:54.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:54 smithi193 sudo[44447]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50503]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50503]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50503]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 sudo[50503]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T01:03:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:54 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: osdmap e22: 4 total, 3 up, 4 in 2024-09-16T01:03:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:03:55.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:55.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:55.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:03:55.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:54 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 sudo[44450]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 sudo[44450]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 sudo[44450]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 sudo[44450]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:55.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43179]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-16T01:03:55.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43179]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:55.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43179]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:55.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43179]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:56.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T01:03:56.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43182]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-16T01:03:56.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43182]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-16T01:03:56.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43182]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 sudo[43182]: pam_unix(sudo:session): session closed for user root 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:03:56.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:03:56.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:03:56.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:56 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:56.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:56 smithi078 ceph-mon[30937]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:56.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:56 smithi078 ceph-mon[30937]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:56 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:56 smithi193 ceph-mon[35701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:56 smithi193 ceph-mon[35701]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:56 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-16T01:03:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:56 smithi148 ceph-mon[35030]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-16T01:03:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:56 smithi148 ceph-mon[35030]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:59.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:03:58 smithi078 ceph-mon[30937]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:03:58 smithi193 ceph-mon[35701]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:03:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:03:58 smithi148 ceph-mon[35030]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:00 smithi078 ceph-mon[30937]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T01:04:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:00 smithi193 ceph-mon[35701]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T01:04:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:00 smithi148 ceph-mon[35030]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-16T01:04:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:02.136 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:01 smithi078 ceph-mon[30937]: Deploying daemon osd.3 on smithi078 2024-09-16T01:04:02.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:01 smithi193 ceph-mon[35701]: Deploying daemon osd.3 on smithi078 2024-09-16T01:04:02.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:01 smithi148 ceph-mon[35030]: Deploying daemon osd.3 on smithi078 2024-09-16T01:04:03.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:02 smithi078 ceph-mon[30937]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:02 smithi193 ceph-mon[35701]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:02 smithi148 ceph-mon[35030]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:05.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:05 smithi078 ceph-mon[30937]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:05 smithi193 ceph-mon[35701]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:05 smithi148 ceph-mon[35030]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:07 smithi078 ceph-mon[30937]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:07.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:07 smithi193 ceph-mon[35701]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:07 smithi148 ceph-mon[35030]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:08.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:08 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:08.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:08 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:08.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:08 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:08.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:08 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:08.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:08 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:08.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:08 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:08 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:08 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:08 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:09.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:09 smithi078 ceph-mon[30937]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:09.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:09 smithi193 ceph-mon[35701]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:09 smithi148 ceph-mon[35030]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:10.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.653+0000 7fd7a0ff9640 1 -- 172.21.15.78:0/156028804 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd7a4062630 con 0x7fd78c0a3260 2024-09-16T01:04:10.655 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-09-16T01:04:10.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 msgr2=0x7fd78c0a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:10.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 0x7fd78c0a5720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4102c60 tx=0x7fd794009110 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 msgr2=0x7fd7a419f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b970 tx=0x7fd79c00bd70 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 shutdown_connections 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd78c0a3260 0x7fd78c0a5720 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a41037c0 0x7fd7a419f650 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7a4102e00 0x7fd7a41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 --2- 172.21.15.78:0/156028804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd7a4101c00 0x7fd7a4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.655+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 >> 172.21.15.78:0/156028804 conn(0x7fd7a40fd390 msgr2=0x7fd7a4106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.656+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 shutdown_connections 2024-09-16T01:04:10.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:04:10.656+0000 7fd7aa0ba640 1 -- 172.21.15.78:0/156028804 wait complete. 2024-09-16T01:04:10.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:10.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:10.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:10.979 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:11.326 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.3.service 2024-09-16T01:04:11.328 INFO:tasks.cephadm:Deploying osd.4 on smithi148 with /dev/vg_nvme/lv_3... 2024-09-16T01:04:11.329 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T01:04:11.585 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:04:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:12 smithi078 ceph-mon[30937]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:12 smithi078 ceph-mon[30937]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T01:04:13.279 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:04:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:04:12.836+0000 7f43d93ab640 -1 osd.3 0 log_to_monitors true 2024-09-16T01:04:13.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:12 smithi193 ceph-mon[35701]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:12 smithi193 ceph-mon[35701]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T01:04:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:12 smithi148 ceph-mon[35030]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:13.373 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:12 smithi148 ceph-mon[35030]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-16T01:04:14.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T01:04:14.166 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_3 2024-09-16T01:04:14.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T01:04:14.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T01:04:14.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:04:14.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:14.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:04:14.172 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-mon[30937]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:04:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:14 smithi193 ceph-mon[35701]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T01:04:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-16T01:04:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: osdmap e23: 4 total, 3 up, 4 in 2024-09-16T01:04:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:04:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:04:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:14 smithi148 ceph-mon[35030]: osdmap e24: 4 total, 3 up, 4 in 2024-09-16T01:04:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:04:14.528 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:04:14.386+0000 7f43d532e640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T01:04:14.528 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:04:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:04:14.391+0000 7f43cd14e640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 -- 172.21.15.148:0/1826655710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928101d10 msgr2=0x7f6928102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 --2- 172.21.15.148:0/1826655710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928101d10 0x7f6928102190 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f691c009900 tx=0x7f691c02a410 comp rx=0 tx=0).stop 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 -- 172.21.15.148:0/1826655710 shutdown_connections 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 --2- 172.21.15.148:0/1826655710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69281026d0 0x7f6928108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 --2- 172.21.15.148:0/1826655710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928101d10 0x7f6928102190 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 --2- 172.21.15.148:0/1826655710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6928100b10 0x7f6928100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 -- 172.21.15.148:0/1826655710 >> 172.21.15.148:0/1826655710 conn(0x7f69280fc2a0 msgr2=0x7f69280fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 -- 172.21.15.148:0/1826655710 shutdown_connections 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.838+0000 7f692cb9d640 1 -- 172.21.15.148:0/1826655710 wait complete. 2024-09-16T01:04:14.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 Processor -- start 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 -- start start 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 0x7f6928198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69281026d0 0x7f692819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f692810d8c0 con 0x7f6928101d10 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f692810d740 con 0x7f6928100b10 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692cb9d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f692810da40 con 0x7f69281026d0 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43588/0 (socket says 172.21.15.148:43588) 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.839+0000 7f692659c640 1 -- 172.21.15.148:0/2157598131 learned_addr learned my addr 172.21.15.148:0/2157598131 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-16T01:04:14.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f6925d9b640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 0x7f6928198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f6926d9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69281026d0 0x7f692819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 -- 172.21.15.148:0/2157598131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69281026d0 msgr2=0x7f692819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69281026d0 0x7f692819f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 -- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 msgr2=0x7f6928198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 0x7f6928198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 -- 172.21.15.148:0/2157598131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692819fa70 con 0x7f6928100b10 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f6925d9b640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 0x7f6928198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692659c640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f691800c970 tx=0x7f691800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918007c10 con 0x7f6928100b10 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6918007db0 con 0x7f6928100b10 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692819fd60 con 0x7f6928100b10 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691800f940 con 0x7f6928100b10 2024-09-16T01:04:14.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.840+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281a01c0 con 0x7f6928100b10 2024-09-16T01:04:14.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.841+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6928107e90 con 0x7f6928100b10 2024-09-16T01:04:14.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.842+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f691800fae0 con 0x7f6928100b10 2024-09-16T01:04:14.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.843+0000 7f69177fe640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 0x7f69100a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:14.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.843+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 3389+0+0 (secure 0 0 0) 0x7f69180ceca0 con 0x7f6928100b10 2024-09-16T01:04:14.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.843+0000 7f6925d9b640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 0x7f69100a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:14.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.844+0000 7f6925d9b640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 0x7f69100a5730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c0095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:04:14.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:14.846+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f691809ce20 con 0x7f6928100b10 2024-09-16T01:04:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:15.025+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6928062630 con 0x7f69100a3270 2024-09-16T01:04:15.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:15.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:15 smithi148 ceph-mon[35030]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' 2024-09-16T01:04:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:15 smithi078 ceph-mon[30937]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:15 smithi193 ceph-mon[35701]: from='osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598]' entity='osd.3' 2024-09-16T01:04:16.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:04:16.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi078 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 8544M 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] boot 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:16.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:16 smithi078 ceph-mon[30937]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi078 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 8544M 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] boot 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:16.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:16.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:16.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:16.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:16 smithi193 ceph-mon[35701]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T01:04:16.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi078 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 8544M 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: osd.3 [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] boot 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: osdmap e25: 4 total, 4 up, 4 in 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:16.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:16 smithi148 ceph-mon[35030]: osdmap e26: 4 total, 4 up, 4 in 2024-09-16T01:04:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:17 smithi078 ceph-mon[30937]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:17 smithi078 ceph-mon[30937]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T01:04:17.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:17 smithi193 ceph-mon[35701]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:17 smithi193 ceph-mon[35701]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T01:04:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:17 smithi148 ceph-mon[35030]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-16T01:04:17.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:17 smithi148 ceph-mon[35030]: osdmap e27: 4 total, 4 up, 4 in 2024-09-16T01:04:18.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/1816949847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]': finished 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 196 KiB/s, 0 objects/s recovering 2024-09-16T01:04:18.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:18 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/55016251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/1816949847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]': finished 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T01:04:18.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:18.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 196 KiB/s, 0 objects/s recovering 2024-09-16T01:04:18.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:18 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/55016251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:18.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: osdmap e28: 4 total, 4 up, 4 in 2024-09-16T01:04:18.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/1816949847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]: dispatch 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8411f2d-a262-4113-95f9-ca6ae72ab967"}]': finished 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: osdmap e29: 5 total, 4 up, 5 in 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 196 KiB/s, 0 objects/s recovering 2024-09-16T01:04:18.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:18 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/55016251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:20 smithi078 ceph-mon[30937]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:04:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:20 smithi193 ceph-mon[35701]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:04:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:20 smithi148 ceph-mon[35030]: pgmap v100: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:04:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:22 smithi078 ceph-mon[30937]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:04:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:22 smithi193 ceph-mon[35701]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:04:23.380 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:22 smithi148 ceph-mon[35030]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:04:25.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:25 smithi078 ceph-mon[30937]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:04:25.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:25 smithi193 ceph-mon[35701]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:04:25.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:25 smithi148 ceph-mon[35030]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:04:26.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T01:04:26.262 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T01:04:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:26.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-16T01:04:26.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:27.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:27 smithi148 ceph-mon[35030]: Deploying daemon osd.4 on smithi148 2024-09-16T01:04:27.263 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:27 smithi148 ceph-mon[35030]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:27 smithi078 ceph-mon[30937]: Deploying daemon osd.4 on smithi148 2024-09-16T01:04:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:27 smithi078 ceph-mon[30937]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:27.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:27 smithi193 ceph-mon[35701]: Deploying daemon osd.4 on smithi148 2024-09-16T01:04:27.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:27 smithi193 ceph-mon[35701]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:29.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:29 smithi148 ceph-mon[35030]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:29.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:29 smithi078 ceph-mon[30937]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:29 smithi193 ceph-mon[35701]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:31.189 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:31 smithi148 ceph-mon[35030]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:31 smithi078 ceph-mon[30937]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:31.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:31 smithi193 ceph-mon[35701]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:33 smithi078 ceph-mon[30937]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:33.290 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:33 smithi148 ceph-mon[35030]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:33.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:33 smithi193 ceph-mon[35701]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:35.225 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:35.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:35.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:35.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.563+0000 7f69177fe640 1 -- 172.21.15.148:0/2157598131 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6928062630 con 0x7f69100a3270 2024-09-16T01:04:35.564 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 4 on host 'smithi148' 2024-09-16T01:04:35.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.565+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 msgr2=0x7f69100a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:35.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.565+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 0x7f69100a5730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c0095f0 comp rx=0 tx=0).stop 2024-09-16T01:04:35.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.565+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 msgr2=0x7f6928197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.565+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f691800c970 tx=0x7f691800ce30 comp rx=0 tx=0).stop 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 shutdown_connections 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f69100a3270 0x7f69100a5730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69281026d0 0x7f692819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6928101d10 0x7f6928198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 --2- 172.21.15.148:0/2157598131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6928100b10 0x7f6928197e40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 >> 172.21.15.148:0/2157598131 conn(0x7f69280fc2a0 msgr2=0x7f6928105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 shutdown_connections 2024-09-16T01:04:35.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:04:35.566+0000 7f692cb9d640 1 -- 172.21.15.148:0/2157598131 wait complete. 2024-09-16T01:04:36.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:36.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.165 DEBUG:teuthology.orchestra.run.smithi148:osd.4> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.4.service 2024-09-16T01:04:36.167 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-09-16T01:04:36.168 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-16T01:04:36.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:36.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:36.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:36.422 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:04:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:37 smithi078 ceph-mon[30937]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:37 smithi193 ceph-mon[35701]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:37.378 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:37 smithi148 ceph-mon[35030]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:38.185 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:38 smithi193 ceph-mon[35701]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.185 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:38 smithi193 ceph-mon[35701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:38 smithi078 ceph-mon[30937]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:38 smithi078 ceph-mon[30937]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:38 smithi148 ceph-mon[35030]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.366 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:38 smithi148 ceph-mon[35030]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-16T01:04:38.366 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:04:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:04:37.911+0000 7fe4e3657640 -1 osd.4 0 log_to_monitors true 2024-09-16T01:04:39.009 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:04:39.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-09-16T01:04:39.285 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:04:39.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T01:04:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T01:04:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:39 smithi193 ceph-mon[35701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T01:04:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T01:04:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-mon[35030]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-16T01:04:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: from='osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: osdmap e30: 5 total, 4 up, 5 in 2024-09-16T01:04:39.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:39.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:39 smithi078 ceph-mon[30937]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:04:39.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 -- 172.21.15.193:0/1247442042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec1037c0 msgr2=0x7ffaec10a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 --2- 172.21.15.193:0/1247442042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec1037c0 0x7ffaec10a050 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009ca0 tx=0x7ffae402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 -- 172.21.15.193:0/1247442042 shutdown_connections 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 --2- 172.21.15.193:0/1247442042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec1037c0 0x7ffaec10a050 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 --2- 172.21.15.193:0/1247442042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec102e00 0x7ffaec103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 --2- 172.21.15.193:0/1247442042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 -- 172.21.15.193:0/1247442042 >> 172.21.15.193:0/1247442042 conn(0x7ffaec0fd390 msgr2=0x7ffaec0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 -- 172.21.15.193:0/1247442042 shutdown_connections 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.763+0000 7ffaf3033640 1 -- 172.21.15.193:0/1247442042 wait complete. 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 Processor -- start 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 -- start start 2024-09-16T01:04:39.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 0x7ffaec074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10d860 con 0x7ffaec101c00 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10d6e0 con 0x7ffaec1037c0 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf3033640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec10d9e0 con 0x7ffaec102e00 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaebfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaebfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:52726/0 (socket says 172.21.15.193:52726) 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaebfff640 1 -- 172.21.15.193:0/568207587 learned_addr learned my addr 172.21.15.193:0/568207587 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf15d0640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 0x7ffaec074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.764+0000 7ffaf0dcf640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:39.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 -- 172.21.15.193:0/568207587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 msgr2=0x7ffaec074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 0x7ffaec074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 -- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 msgr2=0x7ffaec075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 -- 172.21.15.193:0/568207587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec0753e0 con 0x7ffaec102e00 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaf15d0640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 0x7ffaec074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaf0dcf640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaebfff640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffad800c920 tx=0x7ffad800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8007c70 con 0x7ffaec102e00 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad8007e10 con 0x7ffaec102e00 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaec1a1f60 con 0x7ffaec102e00 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8018890 con 0x7ffaec102e00 2024-09-16T01:04:39.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.765+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaec1a23c0 con 0x7ffaec102e00 2024-09-16T01:04:39.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.766+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaec108f30 con 0x7ffaec102e00 2024-09-16T01:04:39.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.767+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ffad8018a70 con 0x7ffaec102e00 2024-09-16T01:04:39.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.768+0000 7ffae9ffb640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 0x7ffadc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:04:39.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.768+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 3821+0+0 (secure 0 0 0) 0x7ffad80ce980 con 0x7ffaec102e00 2024-09-16T01:04:39.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.768+0000 7ffaf0dcf640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 0x7ffadc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:04:39.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.769+0000 7ffaf0dcf640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 0x7ffadc0a5700 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffae00058c0 tx=0x7ffae0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:04:39.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.772+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffad8014030 con 0x7ffaec102e00 2024-09-16T01:04:39.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:04:39.468+0000 7fe4df5da640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T01:04:39.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:04:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:04:39.473+0000 7fe4d73fa640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:04:39.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:04:39.946+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ffaec062630 con 0x7ffadc0a3240 2024-09-16T01:04:40.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:04:40.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T01:04:40.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi148 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='osd.4 ' entity='osd.4' 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:40.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:40.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:40.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi148 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:40.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='osd.4 ' entity='osd.4' 2024-09-16T01:04:40.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:40.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:40.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: osdmap e31: 5 total, 4 up, 5 in 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi148 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='osd.4 ' entity='osd.4' 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:04:40.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:04:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:04:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-16T01:04:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:41.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:41.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:41.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] boot 2024-09-16T01:04:41.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T01:04:41.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:41 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:04:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] boot 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T01:04:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:41 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:41.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:04:41.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: osd.4 [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] boot 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: osdmap e32: 5 total, 5 up, 5 in 2024-09-16T01:04:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:41 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:04:42.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T01:04:42.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/4180059206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]': finished 2024-09-16T01:04:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T01:04:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:42 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:04:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T01:04:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/4180059206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]': finished 2024-09-16T01:04:42.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T01:04:42.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:42 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:04:42.888 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: osdmap e33: 5 total, 5 up, 5 in 2024-09-16T01:04:42.888 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:42.888 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/4180059206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.888 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]: dispatch 2024-09-16T01:04:42.889 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3508afd5-2958-42dd-81c0-5f50be889ec6"}]': finished 2024-09-16T01:04:42.889 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: osdmap e34: 6 total, 5 up, 6 in 2024-09-16T01:04:42.889 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:42 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:04:43.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/258129040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:43.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/258129040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:43.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/258129040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:04:44.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:44 smithi078 ceph-mon[30937]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:44.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:44 smithi193 ceph-mon[35701]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:44 smithi148 ceph-mon[35030]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:47 smithi078 ceph-mon[30937]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:47 smithi193 ceph-mon[35701]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:47 smithi148 ceph-mon[35030]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:49.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:49 smithi193 ceph-mon[35701]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:49 smithi078 ceph-mon[30937]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:49 smithi148 ceph-mon[35030]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:51.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:51 smithi193 ceph-mon[35701]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:51.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:51 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T01:04:51.270 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:51 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:51 smithi078 ceph-mon[30937]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T01:04:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:51 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:51 smithi148 ceph-mon[35030]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:51 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-16T01:04:51.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:51 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:04:52.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:52 smithi078 ceph-mon[30937]: Deploying daemon osd.5 on smithi193 2024-09-16T01:04:52.294 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:52 smithi193 ceph-mon[35701]: Deploying daemon osd.5 on smithi193 2024-09-16T01:04:52.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:52 smithi148 ceph-mon[35030]: Deploying daemon osd.5 on smithi193 2024-09-16T01:04:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:53 smithi078 ceph-mon[30937]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:53.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:53 smithi193 ceph-mon[35701]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:53 smithi148 ceph-mon[35030]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:55 smithi078 ceph-mon[30937]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:55 smithi193 ceph-mon[35701]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:55 smithi148 ceph-mon[35030]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:57 smithi078 ceph-mon[30937]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:57.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:57 smithi193 ceph-mon[35701]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:57 smithi148 ceph-mon[35030]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:58.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:58.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:58.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:58.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:58 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:58.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:58 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:58.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:58 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:58.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:58 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:04:58.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:58 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:58.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:58 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:04:59.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:04:59 smithi078 ceph-mon[30937]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:59.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:04:59 smithi193 ceph-mon[35701]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:04:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:04:59 smithi148 ceph-mon[35030]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:00 smithi193 ceph-mon[35701]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.121+0000 7ffae9ffb640 1 -- 172.21.15.193:0/568207587 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffaec062630 con 0x7ffadc0a3240 2024-09-16T01:05:01.123 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 msgr2=0x7ffadc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 0x7ffadc0a5700 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ffae00058c0 tx=0x7ffae0005850 comp rx=0 tx=0).stop 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 msgr2=0x7ffaec0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffad800c920 tx=0x7ffad800cde0 comp rx=0 tx=0).stop 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 shutdown_connections 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ffadc0a3240 0x7ffadc0a5700 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaec1037c0 0x7ffaec074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:01.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaec102e00 0x7ffaec0745e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:01.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 --2- 172.21.15.193:0/568207587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaec101c00 0x7ffaec075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:01.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 >> 172.21.15.193:0/568207587 conn(0x7ffaec0fd390 msgr2=0x7ffaec106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:01.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 shutdown_connections 2024-09-16T01:05:01.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:01.124+0000 7ffaf3033640 1 -- 172.21.15.193:0/568207587 wait complete. 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:00 smithi148 ceph-mon[35030]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:01.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:01.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:01.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:00 smithi078 ceph-mon[30937]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:01.781 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.5.service 2024-09-16T01:05:01.784 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2024-09-16T01:05:01.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T01:05:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:05:02.045 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:02.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:02.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:01 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:02.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:02.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:02.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:01 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:02.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:02.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:02.159 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:01 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:03.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:02 smithi193 ceph-mon[35701]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:03.088 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:05:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:05:02.928+0000 7f6c8d48b640 -1 osd.5 0 log_to_monitors true 2024-09-16T01:05:03.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:02 smithi148 ceph-mon[35030]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:03.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:02 smithi078 ceph-mon[30937]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:03.993 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:03 smithi078 ceph-mon[30937]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T01:05:04.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:03 smithi148 ceph-mon[35030]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T01:05:04.265 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:03 smithi193 ceph-mon[35701]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-16T01:05:04.597 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:05:04.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-09-16T01:05:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:05:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T01:05:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.143 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:05.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:05.144 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-16T01:05:05.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: osdmap e35: 6 total, 5 up, 6 in 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:05.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 -- 172.21.15.78:0/3000882639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b0102660 msgr2=0x7ff0b0104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/3000882639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b0102660 0x7ff0b0104a50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009900 tx=0x7ff0a002a410 comp rx=0 tx=0).stop 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 -- 172.21.15.78:0/3000882639 shutdown_connections 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/3000882639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b01078c0 0x7ff0b0109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/3000882639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0104f90 0x7ff0b0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/3000882639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b0102660 0x7ff0b0104a50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 -- 172.21.15.78:0/3000882639 >> 172.21.15.78:0/3000882639 conn(0x7ff0b00fc280 msgr2=0x7ff0b00fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 -- 172.21.15.78:0/3000882639 shutdown_connections 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.390+0000 7ff0b51ac640 1 -- 172.21.15.78:0/3000882639 wait complete. 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 Processor -- start 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 -- start start 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0102660 0x7ff0b0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b0104f90 0x7ff0b0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d870 con 0x7ff0b01078c0 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d6f0 con 0x7ff0b0102660 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0b51ac640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d9f0 con 0x7ff0b0104f90 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0af59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0af59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50282/0 (socket says 172.21.15.78:50282) 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.391+0000 7ff0af59e640 1 -- 172.21.15.78:0/2767366178 learned_addr learned my addr 172.21.15.78:0/2767366178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0ae59c640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b0104f90 0x7ff0b0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0102660 0x7ff0b0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 -- 172.21.15.78:0/2767366178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b0104f90 msgr2=0x7ff0b0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b0104f90 0x7ff0b0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 -- 172.21.15.78:0/2767366178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0102660 msgr2=0x7ff0b0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0102660 0x7ff0b0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 -- 172.21.15.78:0/2767366178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b019fb80 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0af59e640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009fd0 tx=0x7ff0a800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8017070 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a8003e50 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b019fe70 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a80106c0 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.392+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b01a02d0 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.394+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b00fdd70 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.394+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff0a8003970 con 0x7ff0b01078c0 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.395+0000 7ff097fff640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 0x7ff0980a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.395+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 0x7ff0980a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.395+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4286+0+0 (secure 0 0 0) 0x7ff0a80ce6a0 con 0x7ff0b01078c0 2024-09-16T01:05:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.396+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 0x7ff0980a5730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000c00 tx=0x7ff0a00315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.400+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff0a809c380 con 0x7ff0b01078c0 2024-09-16T01:05:05.546 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:05:05.245+0000 7f6c8940e640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T01:05:05.546 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:05:05.255+0000 7f6c8122e640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:05:05.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:05.611+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff0b0062630 con 0x7ff0980a3270 2024-09-16T01:05:06.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi193 2024-09-16T01:05:06.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 8544M 2024-09-16T01:05:06.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:05:06.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:06.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:05 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi193 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 8544M 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T01:05:06.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' 2024-09-16T01:05:06.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:06.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:06.182 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:05 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi193 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 8544M 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: osdmap e36: 6 total, 5 up, 6 in 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: osdmap e37: 6 total, 5 up, 6 in 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557]' entity='osd.5' 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:06.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:05 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:07.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] boot 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T01:05:07.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:05:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:05:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] boot 2024-09-16T01:05:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T01:05:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:05:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:05:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-16T01:05:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: osd.5 [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] boot 2024-09-16T01:05:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: osdmap e38: 6 total, 6 up, 6 in 2024-09-16T01:05:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:05:08.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T01:05:08.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:08.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]: dispatch 2024-09-16T01:05:08.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]': finished 2024-09-16T01:05:08.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T01:05:08.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:08 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]: dispatch 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]': finished 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T01:05:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:08 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:08.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: osdmap e39: 6 total, 6 up, 6 in 2024-09-16T01:05:08.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:08.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]: dispatch 2024-09-16T01:05:08.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1160926443' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c2c983a-999e-45fe-bd9e-270546696ff2"}]': finished 2024-09-16T01:05:08.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: osdmap e40: 7 total, 6 up, 7 in 2024-09-16T01:05:08.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:08 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:09.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:09 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3798180953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:09 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3798180953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:09 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3798180953' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:10 smithi078 ceph-mon[30937]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T01:05:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:10 smithi078 ceph-mon[30937]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:10 smithi193 ceph-mon[35701]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T01:05:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:10 smithi193 ceph-mon[35701]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:10 smithi148 ceph-mon[35030]: osdmap e41: 7 total, 6 up, 7 in 2024-09-16T01:05:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:10 smithi148 ceph-mon[35030]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:11 smithi078 ceph-mon[30937]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T01:05:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:11 smithi193 ceph-mon[35701]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T01:05:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:11 smithi148 ceph-mon[35030]: osdmap e42: 7 total, 6 up, 7 in 2024-09-16T01:05:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:12.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:12 smithi078 ceph-mon[30937]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:05:12.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:12 smithi193 ceph-mon[35701]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:05:12.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:12 smithi148 ceph-mon[35030]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-16T01:05:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:15 smithi078 ceph-mon[30937]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:05:15.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:15 smithi193 ceph-mon[35701]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:05:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:15 smithi148 ceph-mon[35030]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:05:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:17 smithi078 ceph-mon[30937]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T01:05:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:17 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T01:05:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:17 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:17 smithi193 ceph-mon[35701]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T01:05:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:17 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T01:05:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:17 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:17 smithi148 ceph-mon[35030]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-16T01:05:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:17 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-16T01:05:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:17 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:18.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:18 smithi078 ceph-mon[30937]: Deploying daemon osd.6 on smithi078 2024-09-16T01:05:18.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:18 smithi193 ceph-mon[35701]: Deploying daemon osd.6 on smithi078 2024-09-16T01:05:18.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:18 smithi148 ceph-mon[35030]: Deploying daemon osd.6 on smithi078 2024-09-16T01:05:19.205 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:19 smithi078 ceph-mon[30937]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:05:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:19 smithi193 ceph-mon[35701]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:05:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:19 smithi148 ceph-mon[35030]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:05:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:21 smithi078 ceph-mon[30937]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T01:05:21.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:21 smithi193 ceph-mon[35701]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T01:05:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:21 smithi148 ceph-mon[35030]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-16T01:05:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:23 smithi078 ceph-mon[30937]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-16T01:05:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:23.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:23 smithi193 ceph-mon[35701]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-16T01:05:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:23 smithi148 ceph-mon[35030]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-16T01:05:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:25.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:25 smithi078 ceph-mon[30937]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:25.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:25 smithi193 ceph-mon[35701]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:25.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:25 smithi148 ceph-mon[35030]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.790+0000 7ff097fff640 1 -- 172.21.15.78:0/2767366178 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff0b0062630 con 0x7ff0980a3270 2024-09-16T01:05:26.792 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2024-09-16T01:05:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 msgr2=0x7ff0980a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 0x7ff0980a5730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000c00 tx=0x7ff0a00315d0 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 msgr2=0x7ff0b019f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009fd0 tx=0x7ff0a800b260 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 shutdown_connections 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff0980a3270 0x7ff0980a5730 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01078c0 0x7ff0b019f510 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0b0104f90 0x7ff0b0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 --2- 172.21.15.78:0/2767366178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0b0102660 0x7ff0b0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.792+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 >> 172.21.15.78:0/2767366178 conn(0x7ff0b00fc280 msgr2=0x7ff0b0102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.793+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 shutdown_connections 2024-09-16T01:05:26.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:05:26.793+0000 7ff0b51ac640 1 -- 172.21.15.78:0/2767366178 wait complete. 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:26 smithi078 ceph-mon[30937]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:26 smithi193 ceph-mon[35701]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:26 smithi148 ceph-mon[35030]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:27.488 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.6.service 2024-09-16T01:05:27.490 INFO:tasks.cephadm:Deploying osd.7 on smithi148 with /dev/vg_nvme/lv_2... 2024-09-16T01:05:27.491 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T01:05:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:05:27.868 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:27.869 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:27.869 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:28.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:28.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:28.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:28.864 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:28 smithi148 ceph-mon[35030]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:28.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:28 smithi078 ceph-mon[30937]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:28 smithi193 ceph-mon[35701]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:29.278 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:05:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:05:29.004+0000 7f2accfea640 -1 osd.6 0 log_to_monitors true 2024-09-16T01:05:29.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:29 smithi078 ceph-mon[30937]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T01:05:30.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:29 smithi193 ceph-mon[35701]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T01:05:30.096 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:29 smithi148 ceph-mon[35030]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-16T01:05:30.383 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T01:05:30.407 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_2 2024-09-16T01:05:30.648 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:31.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T01:05:31.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T01:05:31.093 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:31.094 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 -- 172.21.15.148:0/985484457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 msgr2=0x7f0108102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 --2- 172.21.15.148:0/985484457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108102000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0100009900 tx=0x7f010002a410 comp rx=0 tx=0).stop 2024-09-16T01:05:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 -- 172.21.15.148:0/985484457 shutdown_connections 2024-09-16T01:05:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 --2- 172.21.15.148:0/985484457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 0x7f010810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 --2- 172.21.15.148:0/985484457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 0x7f0108103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 --2- 172.21.15.148:0/985484457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108102000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 -- 172.21.15.148:0/985484457 >> 172.21.15.148:0/985484457 conn(0x7f01080fd390 msgr2=0x7f01080ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 -- 172.21.15.148:0/985484457 shutdown_connections 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.091+0000 7f010d962640 1 -- 172.21.15.148:0/985484457 wait complete. 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 Processor -- start 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 -- start start 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 0x7f01080745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 0x7f0108074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f010810d8e0 con 0x7f01081037c0 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f010810d760 con 0x7f0108101c00 2024-09-16T01:05:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f010d962640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f010810da60 con 0x7f0108102e00 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49852/0 (socket says 172.21.15.148:49852) 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 -- 172.21.15.148:0/2921533636 learned_addr learned my addr 172.21.15.148:0/2921533636 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f01067fc640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 0x7f01080745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f01077fe640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 0x7f0108074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 -- 172.21.15.148:0/2921533636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 msgr2=0x7f01080745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 0x7f01080745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 -- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 msgr2=0x7f0108074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 0x7f0108074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.092+0000 7f0106ffd640 1 -- 172.21.15.148:0/2921533636 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01080753e0 con 0x7f0108101c00 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f0106ffd640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0100031610 tx=0x7f01000043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010003f030 con 0x7f0108101c00 2024-09-16T01:05:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0100003ce0 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01081a2070 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100035070 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.093+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01081a25a0 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.094+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0108109020 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.095+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0100042030 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.096+0000 7f010c960640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 0x7f00f40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.096+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4685+0+0 (secure 0 0 0) 0x7f01000ef130 con 0x7f0108101c00 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.096+0000 7f01067fc640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 0x7f00f40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.096+0000 7f01067fc640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 0x7f00f40a5700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0108075d10 tx=0x7f00f0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.100+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f01000bcc80 con 0x7f0108101c00 2024-09-16T01:05:31.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:31.275+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0108062630 con 0x7f00f40a3240 2024-09-16T01:05:31.278 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:05:30.786+0000 7f2ac8f6d640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T01:05:31.278 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:05:30.794+0000 7f2ac4d95640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: osdmap e43: 7 total, 6 up, 7 in 2024-09-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:05:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:31.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi078 2024-09-16T01:05:31.995 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 5696M 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:31.996 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi078 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 5696M 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:32.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:32.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi078 2024-09-16T01:05:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 5696M 2024-09-16T01:05:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:05:32.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: osdmap e44: 7 total, 6 up, 7 in 2024-09-16T01:05:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:32.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:33.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:05:33.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:05:33.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:33.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] boot 2024-09-16T01:05:33.006 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T01:05:33.007 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:33.007 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:32 smithi148 ceph-mon[35030]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:05:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:05:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] boot 2024-09-16T01:05:33.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T01:05:33.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:33.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:32 smithi078 ceph-mon[30937]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: osd.6 [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] boot 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: osdmap e45: 7 total, 7 up, 7 in 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:05:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:32 smithi193 ceph-mon[35701]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-09-16T01:05:34.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T01:05:34.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/1535518022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]': finished 2024-09-16T01:05:34.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T01:05:34.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:33 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:34.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T01:05:34.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/1535518022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]': finished 2024-09-16T01:05:34.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T01:05:34.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:33 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: osdmap e46: 7 total, 7 up, 7 in 2024-09-16T01:05:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/1535518022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]: dispatch 2024-09-16T01:05:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9855f37-2fd8-44a6-a6d9-80cf978e357a"}]': finished 2024-09-16T01:05:34.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: osdmap e47: 8 total, 7 up, 8 in 2024-09-16T01:05:34.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:33 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:34 smithi148 ceph-mon[35030]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:35.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:34 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/3592567380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:35.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:34 smithi078 ceph-mon[30937]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:35.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:34 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/3592567380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:34 smithi193 ceph-mon[35701]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:34 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/3592567380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:37.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:37 smithi078 ceph-mon[30937]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:37 smithi193 ceph-mon[35701]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:37 smithi148 ceph-mon[35030]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:39 smithi078 ceph-mon[30937]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:39.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:39 smithi193 ceph-mon[35701]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:39.367 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:39 smithi148 ceph-mon[35030]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:41 smithi148 ceph-mon[35030]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:41.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:41 smithi078 ceph-mon[30937]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:41 smithi193 ceph-mon[35701]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:42.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:42 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T01:05:42.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:42 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:42.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:42 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T01:05:42.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:42 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:42.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:42 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-16T01:05:42.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:42 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:43 smithi078 ceph-mon[30937]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:43 smithi078 ceph-mon[30937]: Deploying daemon osd.7 on smithi148 2024-09-16T01:05:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:43 smithi193 ceph-mon[35701]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:43 smithi193 ceph-mon[35701]: Deploying daemon osd.7 on smithi148 2024-09-16T01:05:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:43 smithi148 ceph-mon[35030]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:43 smithi148 ceph-mon[35030]: Deploying daemon osd.7 on smithi148 2024-09-16T01:05:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:45 smithi148 ceph-mon[35030]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:45 smithi078 ceph-mon[30937]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:45.295 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:45 smithi193 ceph-mon[35701]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:47.270 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:47 smithi148 ceph-mon[35030]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:47 smithi078 ceph-mon[30937]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:47 smithi193 ceph-mon[35701]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:49.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:49 smithi148 ceph-mon[35030]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:49.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:49.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:49.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:49 smithi078 ceph-mon[30937]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:49.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:49 smithi193 ceph-mon[35701]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:51.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:51 smithi148 ceph-mon[35030]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:51 smithi193 ceph-mon[35701]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:51.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.510+0000 7f010c960640 1 -- 172.21.15.148:0/2921533636 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0108062630 con 0x7f00f40a3240 2024-09-16T01:05:51.511 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 7 on host 'smithi148' 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 msgr2=0x7f00f40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 0x7f00f40a5700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0108075d10 tx=0x7f00f0008040 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 msgr2=0x7f0108075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0100031610 tx=0x7f01000043b0 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 shutdown_connections 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f00f40a3240 0x7f00f40a5700 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01081037c0 0x7f0108074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0108102e00 0x7f01080745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 --2- 172.21.15.148:0/2921533636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0108101c00 0x7f0108075f90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 >> 172.21.15.148:0/2921533636 conn(0x7f01080fd390 msgr2=0x7f01081061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:51.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 shutdown_connections 2024-09-16T01:05:51.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:05:51.513+0000 7f010d962640 1 -- 172.21.15.148:0/2921533636 wait complete. 2024-09-16T01:05:51.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:51 smithi078 ceph-mon[30937]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:52.246 DEBUG:teuthology.orchestra.run.smithi148:osd.7> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.7.service 2024-09-16T01:05:52.248 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-09-16T01:05:52.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-16T01:05:52.504 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:05:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:52 smithi193 ceph-mon[35701]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:52 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.764 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.764 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:52 smithi148 ceph-mon[35030]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:52.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.765 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:52 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:52 smithi078 ceph-mon[30937]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:52 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:54.048 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:05:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:05:54.003+0000 7f9fbac41640 -1 osd.7 0 log_to_monitors true 2024-09-16T01:05:54.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:54 smithi193 ceph-mon[35701]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:54.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:54 smithi193 ceph-mon[35701]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:54 smithi148 ceph-mon[35030]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:54 smithi148 ceph-mon[35030]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:54.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:54 smithi078 ceph-mon[30937]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:54.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:54 smithi078 ceph-mon[30937]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-16T01:05:55.096 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:05:55.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-09-16T01:05:55.146 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:55.146 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T01:05:55.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T01:05:55.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:55.147 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:55 smithi193 ceph-mon[35701]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.372 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:05:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T01:05:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T01:05:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-mon[35030]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-16T01:05:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: from='osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: osdmap e48: 8 total, 7 up, 8 in 2024-09-16T01:05:55.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:55.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:55 smithi078 ceph-mon[30937]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:05:55.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 -- 172.21.15.193:0/1781694262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2041037c0 msgr2=0x7fa20410a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:55.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/1781694262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2041037c0 0x7fa20410a050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009ca0 tx=0x7fa1fc02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 -- 172.21.15.193:0/1781694262 shutdown_connections 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/1781694262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2041037c0 0x7fa20410a050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/1781694262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa204102e00 0x7fa204103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/1781694262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204101c00 0x7fa204102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 -- 172.21.15.193:0/1781694262 >> 172.21.15.193:0/1781694262 conn(0x7fa2040fd390 msgr2=0x7fa2040ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 -- 172.21.15.193:0/1781694262 shutdown_connections 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.837+0000 7fa20b9ae640 1 -- 172.21.15.193:0/1781694262 wait complete. 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 Processor -- start 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 -- start start 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204102e00 0x7fa2040745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2041037c0 0x7fa204074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410d880 con 0x7fa2041037c0 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410d700 con 0x7fa204102e00 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20b9ae640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa20410da00 con 0x7fa204101c00 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:37662/0 (socket says 172.21.15.193:37662) 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 -- 172.21.15.193:0/3392795259 learned_addr learned my addr 172.21.15.193:0/3392795259 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa208f49640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204102e00 0x7fa2040745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 -- 172.21.15.193:0/3392795259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204102e00 msgr2=0x7fa2040745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204102e00 0x7fa2040745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 -- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2041037c0 msgr2=0x7fa204074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2041037c0 0x7fa204074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:05:55.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.838+0000 7fa20974a640 1 -- 172.21.15.193:0/3392795259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2040753e0 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa20974a640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9c0 tx=0x7fa20000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20000cd80 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2000035f0 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2041a1fb0 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200010690 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.839+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2041a2410 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.840+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa204108f80 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.841+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa2000037a0 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.842+0000 7fa1fa7fc640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 0x7fa1f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.842+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5150+0+0 (secure 0 0 0) 0x7fa2000cf110 con 0x7fa204101c00 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.842+0000 7fa208f49640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 0x7fa1f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:05:55.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.842+0000 7fa208f49640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 0x7fa1f00a5700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa204075d10 tx=0x7fa1f400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:05:55.867 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:55.845+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa20009ca90 con 0x7fa204101c00 2024-09-16T01:05:55.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:05:55.477+0000 7f9fb6bc4640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T01:05:55.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:05:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:05:55.489+0000 7f9fae9e4640 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:05:56.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:05:56.019+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa204062630 con 0x7fa1f00a3240 2024-09-16T01:05:56.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:05:56.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T01:05:56.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='osd.7 ' entity='osd.7' 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:56.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:56 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='osd.7 ' entity='osd.7' 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:56.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:56 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: osdmap e49: 8 total, 7 up, 8 in 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: osdmap e50: 8 total, 7 up, 8 in 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='osd.7 ' entity='osd.7' 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:05:56.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:56 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:05:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi148 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:57.731 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:57.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] boot 2024-09-16T01:05:57.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T01:05:57.732 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:57.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi148 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] boot 2024-09-16T01:05:57.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi148 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-16T01:05:57.737 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-09-16T01:05:57.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:05:57.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:57.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: osd.7 [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] boot 2024-09-16T01:05:57.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: osdmap e51: 8 total, 8 up, 8 in 2024-09-16T01:05:57.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:05:58.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:58 smithi193 ceph-mon[35701]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T01:05:58.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:58 smithi193 ceph-mon[35701]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:05:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:58 smithi078 ceph-mon[30937]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T01:05:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:58 smithi078 ceph-mon[30937]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:05:58.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:58 smithi148 ceph-mon[35030]: osdmap e52: 8 total, 8 up, 8 in 2024-09-16T01:05:58.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:58 smithi148 ceph-mon[35030]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/4156046539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]': finished 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:05:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:05:59 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/3623935453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/4156046539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]': finished 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:05:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:05:59 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/3623935453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:05:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: osdmap e53: 8 total, 8 up, 8 in 2024-09-16T01:05:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/4156046539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]: dispatch 2024-09-16T01:05:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be96be1e-a654-46f9-a1aa-d2ce999d7a67"}]': finished 2024-09-16T01:05:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: osdmap e54: 9 total, 8 up, 9 in 2024-09-16T01:05:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:05:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:05:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/3623935453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:00.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:00 smithi078 ceph-mon[30937]: osdmap e55: 9 total, 8 up, 9 in 2024-09-16T01:06:00.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:00.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:00 smithi078 ceph-mon[30937]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:00 smithi193 ceph-mon[35701]: osdmap e55: 9 total, 8 up, 9 in 2024-09-16T01:06:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:00 smithi193 ceph-mon[35701]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:00.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:00 smithi148 ceph-mon[35030]: osdmap e55: 9 total, 8 up, 9 in 2024-09-16T01:06:00.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:00.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:00 smithi148 ceph-mon[35030]: pgmap v176: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:03.271 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:03 smithi193 ceph-mon[35701]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:03.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:03 smithi078 ceph-mon[30937]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:03.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:03 smithi148 ceph-mon[35030]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:05.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:05 smithi193 ceph-mon[35701]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:05 smithi078 ceph-mon[30937]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:05 smithi148 ceph-mon[35030]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:07 smithi078 ceph-mon[30937]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T01:06:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:07 smithi193 ceph-mon[35701]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T01:06:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:07 smithi148 ceph-mon[35030]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-16T01:06:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:08.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:08 smithi078 ceph-mon[30937]: Deploying daemon osd.8 on smithi193 2024-09-16T01:06:08.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:08 smithi193 ceph-mon[35701]: Deploying daemon osd.8 on smithi193 2024-09-16T01:06:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:08 smithi148 ceph-mon[35030]: Deploying daemon osd.8 on smithi193 2024-09-16T01:06:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:09 smithi078 ceph-mon[30937]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:09.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:09 smithi193 ceph-mon[35701]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:09 smithi148 ceph-mon[35030]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:11 smithi193 ceph-mon[35701]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:11 smithi148 ceph-mon[35030]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:11 smithi078 ceph-mon[30937]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:13 smithi193 ceph-mon[35701]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:13 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:13 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:13 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:13 smithi148 ceph-mon[35030]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:13 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:13 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:13 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:13 smithi078 ceph-mon[30937]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:13 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:13 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:13 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:15 smithi193 ceph-mon[35701]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:15 smithi148 ceph-mon[35030]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:15 smithi078 ceph-mon[30937]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:16.613 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.610+0000 7fa1fa7fc640 1 -- 172.21.15.193:0/3392795259 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa204062630 con 0x7fa1f00a3240 2024-09-16T01:06:16.613 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 msgr2=0x7fa1f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 0x7fa1f00a5700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa204075d10 tx=0x7fa1f400b390 comp rx=0 tx=0).stop 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 msgr2=0x7fa204075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9c0 tx=0x7fa20000ee80 comp rx=0 tx=0).stop 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 shutdown_connections 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa1f00a3240 0x7fa1f00a5700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:16.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2041037c0 0x7fa204074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:16.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa204102e00 0x7fa2040745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:16.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 --2- 172.21.15.193:0/3392795259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa204101c00 0x7fa204075f90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:16.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 >> 172.21.15.193:0/3392795259 conn(0x7fa2040fd390 msgr2=0x7fa204106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:06:16.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 shutdown_connections 2024-09-16T01:06:16.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:06:16.613+0000 7fa20b9ae640 1 -- 172.21.15.193:0/3392795259 wait complete. 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:16.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:17.294 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.8.service 2024-09-16T01:06:17.336 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2024-09-16T01:06:17.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T01:06:17.588 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:06:18.985 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:06:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:06:18.969+0000 7f15fdf0a640 -1 osd.8 0 log_to_monitors true 2024-09-16T01:06:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:19 smithi078 ceph-mon[30937]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:19 smithi078 ceph-mon[30937]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T01:06:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:19 smithi193 ceph-mon[35701]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:19 smithi193 ceph-mon[35701]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T01:06:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:19 smithi148 ceph-mon[35030]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:19 smithi148 ceph-mon[35030]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-16T01:06:20.257 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:06:20.025+0000 7f15f9e8d640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T01:06:20.258 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:06:20.034+0000 7f15f5cb5640 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:06:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-mon[35701]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T01:06:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-mon[35701]: osdmap e56: 9 total, 8 up, 9 in 2024-09-16T01:06:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-mon[35701]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:06:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:20 smithi193 ceph-mon[35701]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:06:20.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:20 smithi078 ceph-mon[30937]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T01:06:20.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:20 smithi078 ceph-mon[30937]: osdmap e56: 9 total, 8 up, 9 in 2024-09-16T01:06:20.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:20 smithi078 ceph-mon[30937]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:06:20.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:20.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:20 smithi078 ceph-mon[30937]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:06:20.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:20 smithi148 ceph-mon[35030]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-16T01:06:20.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:20 smithi148 ceph-mon[35030]: osdmap e56: 9 total, 8 up, 9 in 2024-09-16T01:06:20.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:20 smithi148 ceph-mon[35030]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:06:20.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:20.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:20 smithi148 ceph-mon[35030]: from='osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816]' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:06:20.455 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:06:20.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2024-09-16T01:06:20.727 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:06:21.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: osdmap e57: 9 total, 8 up, 9 in 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] boot 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:21.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:21.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:21 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 -- 172.21.15.78:0/2902664469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c101c00 msgr2=0x7fe25c102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 --2- 172.21.15.78:0/2902664469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c101c00 0x7fe25c102000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe258009900 tx=0x7fe25802a410 comp rx=0 tx=0).stop 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 -- 172.21.15.78:0/2902664469 shutdown_connections 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 --2- 172.21.15.78:0/2902664469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c1037c0 0x7fe25c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 --2- 172.21.15.78:0/2902664469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c102e00 0x7fe25c103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 --2- 172.21.15.78:0/2902664469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c101c00 0x7fe25c102000 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 -- 172.21.15.78:0/2902664469 >> 172.21.15.78:0/2902664469 conn(0x7fe25c0fd390 msgr2=0x7fe25c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 -- 172.21.15.78:0/2902664469 shutdown_connections 2024-09-16T01:06:21.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.197+0000 7fe264316640 1 -- 172.21.15.78:0/2902664469 wait complete. 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 Processor -- start 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 -- start start 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c101c00 0x7fe25c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c102e00 0x7fe25c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10d960 con 0x7fe25c1037c0 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10d7e0 con 0x7fe25c102e00 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe264316640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10dae0 con 0x7fe25c101c00 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe2628b3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:21.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe2628b3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56206/0 (socket says 172.21.15.78:56206) 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe2628b3640 1 -- 172.21.15.78:0/3794102855 learned_addr learned my addr 172.21.15.78:0/3794102855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe2618b1640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c102e00 0x7fe25c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.198+0000 7fe2620b2640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c101c00 0x7fe25c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 -- 172.21.15.78:0/3794102855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c101c00 msgr2=0x7fe25c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c101c00 0x7fe25c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 -- 172.21.15.78:0/3794102855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c102e00 msgr2=0x7fe25c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c102e00 0x7fe25c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 -- 172.21.15.78:0/3794102855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25c19fc80 con 0x7fe25c1037c0 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2628b3640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b970 tx=0x7fe25400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:06:21.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254019070 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25c19ff70 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe25400cb40 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254007650 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.199+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25c1a0450 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.200+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe25c1090a0 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.201+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe25401c070 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.202+0000 7fe2537fe640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 0x7fe2480a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.202+0000 7fe2620b2640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 0x7fe2480a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.202+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5858+0+0 (secure 0 0 0) 0x7fe2540ceac0 con 0x7fe25c1037c0 2024-09-16T01:06:21.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.203+0000 7fe2620b2640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 0x7fe2480a5730 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe258006010 tx=0x7fe258000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:06:21.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.206+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe25409c0d0 con 0x7fe25c1037c0 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: osdmap e57: 9 total, 8 up, 9 in 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] boot 2024-09-16T01:06:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:21.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:21 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:21.381+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe25c062630 con 0x7fe2480a3270 2024-09-16T01:06:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: osdmap e57: 9 total, 8 up, 9 in 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: osd.8 [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] boot 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: osdmap e58: 9 total, 9 up, 9 in 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:21.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:21 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:22.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:06:22.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:06:22.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi193 2024-09-16T01:06:22.118 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 5696M 2024-09-16T01:06:22.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: osdmap e59: 9 total, 9 up, 9 in 2024-09-16T01:06:22.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:22.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:22.119 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi193 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 5696M 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: osdmap e59: 9 total, 9 up, 9 in 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:22.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:22.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:22.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:06:22.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi193 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 5696M 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: osdmap e59: 9 total, 9 up, 9 in 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:22.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:23.077 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:23 smithi078 ceph-mon[30937]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:23.077 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:23 smithi078 ceph-mon[30937]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:23.077 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:23 smithi078 ceph-mon[30937]: osdmap e60: 9 total, 9 up, 9 in 2024-09-16T01:06:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:23 smithi193 ceph-mon[35701]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:23 smithi193 ceph-mon[35701]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:23 smithi193 ceph-mon[35701]: osdmap e60: 9 total, 9 up, 9 in 2024-09-16T01:06:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:23 smithi148 ceph-mon[35030]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:23 smithi148 ceph-mon[35030]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:23 smithi148 ceph-mon[35030]: osdmap e60: 9 total, 9 up, 9 in 2024-09-16T01:06:24.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: osdmap e61: 9 total, 9 up, 9 in 2024-09-16T01:06:24.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]: dispatch 2024-09-16T01:06:24.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]': finished 2024-09-16T01:06:24.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: osdmap e62: 10 total, 9 up, 10 in 2024-09-16T01:06:24.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:24.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:24 smithi078 ceph-mon[30937]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: osdmap e61: 9 total, 9 up, 9 in 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]: dispatch 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]': finished 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: osdmap e62: 10 total, 9 up, 10 in 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:24.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:24 smithi193 ceph-mon[35701]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:24.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: osdmap e61: 9 total, 9 up, 9 in 2024-09-16T01:06:24.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]: dispatch 2024-09-16T01:06:24.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1352501764' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc303551-6913-47a4-ade8-be306c49a4a3"}]': finished 2024-09-16T01:06:24.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: osdmap e62: 10 total, 9 up, 10 in 2024-09-16T01:06:24.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:24.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:24 smithi148 ceph-mon[35030]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:25.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:25 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/886753683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:25 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/886753683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:25 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/886753683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:26.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:26 smithi078 ceph-mon[30937]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:26.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:26 smithi193 ceph-mon[35701]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:26.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:26 smithi148 ceph-mon[35030]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:29.190 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:29 smithi078 ceph-mon[30937]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:06:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:29 smithi193 ceph-mon[35701]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:06:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:29 smithi148 ceph-mon[35030]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-16T01:06:31.269 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:31 smithi078 ceph-mon[30937]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:06:31.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:31 smithi193 ceph-mon[35701]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:06:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:31 smithi148 ceph-mon[35030]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-16T01:06:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:33 smithi078 ceph-mon[30937]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:06:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:33 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T01:06:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:33 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:33 smithi193 ceph-mon[35701]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:06:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:33 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T01:06:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:33 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:33 smithi148 ceph-mon[35030]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-16T01:06:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:33 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-16T01:06:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:33 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:34 smithi193 ceph-mon[35701]: Deploying daemon osd.9 on smithi078 2024-09-16T01:06:34.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:34 smithi078 ceph-mon[30937]: Deploying daemon osd.9 on smithi078 2024-09-16T01:06:34.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:34 smithi148 ceph-mon[35030]: Deploying daemon osd.9 on smithi078 2024-09-16T01:06:35.316 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:35 smithi193 ceph-mon[35701]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T01:06:35.391 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:35 smithi148 ceph-mon[35030]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T01:06:35.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:35 smithi078 ceph-mon[30937]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-16T01:06:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:37 smithi078 ceph-mon[30937]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:37 smithi193 ceph-mon[35701]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:37 smithi148 ceph-mon[35030]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:39 smithi148 ceph-mon[35030]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:39 smithi078 ceph-mon[30937]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:39.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:39 smithi193 ceph-mon[35701]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:06:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:41.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:41 smithi078 ceph-mon[30937]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:41 smithi148 ceph-mon[35030]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:41 smithi193 ceph-mon[35701]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:42.066 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2024-09-16T01:06:42.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.064+0000 7fe2537fe640 1 -- 172.21.15.78:0/3794102855 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe25c062630 con 0x7fe2480a3270 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 msgr2=0x7fe2480a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 0x7fe2480a5730 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe258006010 tx=0x7fe258000bc0 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 msgr2=0x7fe25c19f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b970 tx=0x7fe25400be30 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 shutdown_connections 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe2480a3270 0x7fe2480a5730 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe25c1037c0 0x7fe25c19f610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25c102e00 0x7fe25c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 --2- 172.21.15.78:0/3794102855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe25c101c00 0x7fe25c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:42.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 >> 172.21.15.78:0/3794102855 conn(0x7fe25c0fd390 msgr2=0x7fe25c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:06:42.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 shutdown_connections 2024-09-16T01:06:42.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:06:42.066+0000 7fe264316640 1 -- 172.21.15.78:0/3794102855 wait complete. 2024-09-16T01:06:42.890 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.9.service 2024-09-16T01:06:42.893 INFO:tasks.cephadm:Deploying osd.10 on smithi148 with /dev/vg_nvme/lv_1... 2024-09-16T01:06:42.893 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T01:06:43.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:43 smithi078 ceph-mon[30937]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:43.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.148 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:43 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:06:43.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:43 smithi148 ceph-mon[35030]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:43.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.271 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:43 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:43 smithi193 ceph-mon[35701]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:43 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:44.693 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:06:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:06:44.611+0000 7fef3fe44640 -1 osd.9 0 log_to_monitors true 2024-09-16T01:06:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:45 smithi148 ceph-mon[35030]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:45.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:45 smithi148 ceph-mon[35030]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T01:06:45.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:45 smithi078 ceph-mon[30937]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:45.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:45 smithi078 ceph-mon[30937]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T01:06:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:45 smithi193 ceph-mon[35701]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:45 smithi193 ceph-mon[35701]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-16T01:06:45.580 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T01:06:45.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_1 2024-09-16T01:06:45.860 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:06:46.254 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:46 smithi148 ceph-mon[35030]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T01:06:46.254 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:46 smithi148 ceph-mon[35030]: osdmap e63: 10 total, 9 up, 10 in 2024-09-16T01:06:46.254 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:46 smithi148 ceph-mon[35030]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:06:46.254 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:46.254 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:46 smithi148 ceph-mon[35030]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:06:46.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 -- 172.21.15.148:0/4268335050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7578101d10 msgr2=0x7f7578102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:46.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 --2- 172.21.15.148:0/4268335050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7578101d10 0x7f7578102190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7568009900 tx=0x7f756802a410 comp rx=0 tx=0).stop 2024-09-16T01:06:46.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 -- 172.21.15.148:0/4268335050 shutdown_connections 2024-09-16T01:06:46.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 --2- 172.21.15.148:0/4268335050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75781026d0 0x7f7578108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:46.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 --2- 172.21.15.148:0/4268335050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7578101d10 0x7f7578102190 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 --2- 172.21.15.148:0/4268335050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 0x7f7578100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 -- 172.21.15.148:0/4268335050 >> 172.21.15.148:0/4268335050 conn(0x7f75780fc2a0 msgr2=0x7f75780fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 -- 172.21.15.148:0/4268335050 shutdown_connections 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.273+0000 7f7580413640 1 -- 172.21.15.148:0/4268335050 wait complete. 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 Processor -- start 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 -- start start 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 0x7f7578197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7578101d10 0x7f7578198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:46.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f757810c790 con 0x7f7578101d10 2024-09-16T01:06:46.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f757810c610 con 0x7f75781026d0 2024-09-16T01:06:46.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f7580413640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f757810c910 con 0x7f7578100b10 2024-09-16T01:06:46.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38076/0 (socket says 172.21.15.148:38076) 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 -- 172.21.15.148:0/1292609293 learned_addr learned my addr 172.21.15.148:0/1292609293 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757d9ae640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7578101d10 0x7f7578198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e1af640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 0x7f7578197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 -- 172.21.15.148:0/1292609293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 msgr2=0x7f7578197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 0x7f7578197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 -- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7578101d10 msgr2=0x7f7578198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7578101d10 0x7f7578198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:06:46.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.274+0000 7f757e9b0640 1 -- 172.21.15.148:0/1292609293 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757819fb40 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f757e9b0640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f757000eb10 tx=0x7f757000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570004b20 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7570004cc0 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757819fe30 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570010760 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.275+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75781a0290 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.276+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75781a0520 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.277+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7570010900 con 0x7f75781026d0 2024-09-16T01:06:46.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.278+0000 7f756f7fe640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 0x7f75640a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:06:46.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.278+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5981+0+0 (secure 0 0 0) 0x7f75700cf7f0 con 0x7f75781026d0 2024-09-16T01:06:46.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.278+0000 7f757e1af640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 0x7f75640a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:06:46.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.279+0000 7f757e1af640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 0x7f75640a5730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7578101b70 tx=0x7f7574005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:06:46.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.282+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75700541b0 con 0x7f75781026d0 2024-09-16T01:06:46.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:06:46.461+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f75781a06f0 con 0x7f75640a3270 2024-09-16T01:06:46.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-mon[30937]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T01:06:46.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-mon[30937]: osdmap e63: 10 total, 9 up, 10 in 2024-09-16T01:06:46.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-mon[30937]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:06:46.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:46.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-mon[30937]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:06:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:46 smithi193 ceph-mon[35701]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-16T01:06:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:46 smithi193 ceph-mon[35701]: osdmap e63: 10 total, 9 up, 10 in 2024-09-16T01:06:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:46 smithi193 ceph-mon[35701]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-09-16T01:06:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:46 smithi193 ceph-mon[35701]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-09-16T01:06:46.778 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:06:46.497+0000 7fef3bdc7640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T01:06:46.778 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:06:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:06:46.507+0000 7fef33be7640 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: osdmap e64: 10 total, 9 up, 10 in 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.138 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:47.138 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:47.138 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.138 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:47 smithi148 ceph-mon[35030]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: osdmap e64: 10 total, 9 up, 10 in 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.448 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:47.449 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:47 smithi193 ceph-mon[35701]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' 2024-09-16T01:06:47.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: osdmap e64: 10 total, 9 up, 10 in 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:06:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:47.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:47 smithi078 ceph-mon[30937]: from='osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790]' entity='osd.9' 2024-09-16T01:06:48.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:06:48.108 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi078 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 4272M 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] boot 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: osdmap e65: 10 total, 10 up, 10 in 2024-09-16T01:06:48.109 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:48 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:48.343 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi078 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 4272M 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] boot 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: osdmap e65: 10 total, 10 up, 10 in 2024-09-16T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:48 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi078 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 4272M 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: osd.9 [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] boot 2024-09-16T01:06:48.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: osdmap e65: 10 total, 10 up, 10 in 2024-09-16T01:06:48.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:48 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:06:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: osdmap e66: 10 total, 10 up, 10 in 2024-09-16T01:06:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/1435082004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]': finished 2024-09-16T01:06:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: osdmap e67: 11 total, 10 up, 11 in 2024-09-16T01:06:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:49 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:06:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: osdmap e66: 10 total, 10 up, 10 in 2024-09-16T01:06:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/1435082004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]': finished 2024-09-16T01:06:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: osdmap e67: 11 total, 10 up, 11 in 2024-09-16T01:06:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:49 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: osdmap e66: 10 total, 10 up, 10 in 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/1435082004' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]: dispatch 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae7b8367-0244-4150-97f6-8ff99149e440"}]': finished 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: osdmap e67: 11 total, 10 up, 11 in 2024-09-16T01:06:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:49 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:06:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.148:0/706823110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:50.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.148:0/706823110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.148:0/706823110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:06:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:51 smithi148 ceph-mon[35030]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:51.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:51 smithi078 ceph-mon[30937]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:51 smithi193 ceph-mon[35701]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:53.244 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:53 smithi148 ceph-mon[35030]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:53.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:53 smithi078 ceph-mon[30937]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:53 smithi193 ceph-mon[35701]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:55 smithi148 ceph-mon[35030]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:55 smithi078 ceph-mon[30937]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:55 smithi193 ceph-mon[35701]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:56.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:56 smithi148 ceph-mon[35030]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:56.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:56 smithi078 ceph-mon[30937]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:56 smithi193 ceph-mon[35701]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:57.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T01:06:57.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:57.384 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:57 smithi148 ceph-mon[35030]: Deploying daemon osd.10 on smithi148 2024-09-16T01:06:57.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T01:06:57.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:57.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:57 smithi078 ceph-mon[30937]: Deploying daemon osd.10 on smithi148 2024-09-16T01:06:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-16T01:06:57.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:06:57.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:57 smithi193 ceph-mon[35701]: Deploying daemon osd.10 on smithi148 2024-09-16T01:06:58.355 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:06:58 smithi148 ceph-mon[35030]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:06:58 smithi078 ceph-mon[30937]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:06:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:06:58 smithi193 ceph-mon[35701]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:01 smithi078 ceph-mon[30937]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:01 smithi193 ceph-mon[35701]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:01.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:01 smithi148 ceph-mon[35030]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:03 smithi193 ceph-mon[35701]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:03 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:03 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:03 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:03.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:03 smithi148 ceph-mon[35030]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:03.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:03 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:03.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:03 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:03.353 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:03 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:03 smithi078 ceph-mon[30937]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:03 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:03 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:03 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:05 smithi193 ceph-mon[35701]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:05.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:05 smithi078 ceph-mon[30937]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:05.367 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:05 smithi148 ceph-mon[35030]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:06.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.361+0000 7f756f7fe640 1 -- 172.21.15.148:0/1292609293 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f75781a06f0 con 0x7f75640a3270 2024-09-16T01:07:06.362 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 10 on host 'smithi148' 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 msgr2=0x7f75640a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 0x7f75640a5730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7578101b70 tx=0x7f7574005f90 comp rx=0 tx=0).stop 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 msgr2=0x7f757819f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f757000eb10 tx=0x7f757000c2d0 comp rx=0 tx=0).stop 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 shutdown_connections 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f75640a3270 0x7f75640a5730 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f75781026d0 0x7f757819f4d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7578101d10 0x7f7578198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:06.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.363+0000 7f7580413640 1 --2- 172.21.15.148:0/1292609293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7578100b10 0x7f7578197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:06.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.364+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 >> 172.21.15.148:0/1292609293 conn(0x7f75780fc2a0 msgr2=0x7f757806a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:06.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.364+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 shutdown_connections 2024-09-16T01:07:06.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T01:07:06.364+0000 7f7580413640 1 -- 172.21.15.148:0/1292609293 wait complete. 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:06.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:06.926 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:06 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.026 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:06 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:07.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:06 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:07.053 DEBUG:teuthology.orchestra.run.smithi148:osd.10> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.10.service 2024-09-16T01:07:07.056 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-09-16T01:07:07.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-16T01:07:07.311 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:07:09.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:09 smithi193 ceph-mon[35701]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:09 smithi078 ceph-mon[30937]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:09 smithi148 ceph-mon[35030]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:07:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:07:09.099+0000 7fe9f6641640 -1 osd.10 0 log_to_monitors true 2024-09-16T01:07:10.047 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-09-16T01:07:10.073 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-09-16T01:07:10.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:10.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.323 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:10.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-16T01:07:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:10.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 -- 172.21.15.193:0/1409779850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2101026d0 msgr2=0x7fc210108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 --2- 172.21.15.193:0/1409779850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2101026d0 0x7fc210108f60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc208009ca0 tx=0x7fc20802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 -- 172.21.15.193:0/1409779850 shutdown_connections 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 --2- 172.21.15.193:0/1409779850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2101026d0 0x7fc210108f60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 --2- 172.21.15.193:0/1409779850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc210101d10 0x7fc210102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 --2- 172.21.15.193:0/1409779850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 0x7fc210100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 -- 172.21.15.193:0/1409779850 >> 172.21.15.193:0/1409779850 conn(0x7fc2100fc2a0 msgr2=0x7fc2100fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 -- 172.21.15.193:0/1409779850 shutdown_connections 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.829+0000 7fc217787640 1 -- 172.21.15.193:0/1409779850 wait complete. 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 Processor -- start 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 -- start start 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 0x7fc210197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:10.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2101026d0 0x7fc21019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc21010c810 con 0x7fc2101026d0 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc21010c690 con 0x7fc210100b10 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc217787640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc21010c990 con 0x7fc210101d10 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc214d22640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc214d22640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:34960/0 (socket says 172.21.15.193:34960) 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc214d22640 1 -- 172.21.15.193:0/964726502 learned_addr learned my addr 172.21.15.193:0/964726502 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc215523640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 0x7fc210197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.830+0000 7fc215d24640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2101026d0 0x7fc21019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 -- 172.21.15.193:0/964726502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 msgr2=0x7fc210197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:10.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 0x7fc210197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 -- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2101026d0 msgr2=0x7fc21019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2101026d0 0x7fc21019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 -- 172.21.15.193:0/964726502 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21019fb20 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc214d22640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc20400d950 tx=0x7fc20400dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc204017070 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21019fe10 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc20400de80 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20400f620 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.831+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101a0390 con 0x7fc210101d10 2024-09-16T01:07:10.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.832+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2101a05c0 con 0x7fc210101d10 2024-09-16T01:07:10.837 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.833+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fc204009c30 con 0x7fc210101d10 2024-09-16T01:07:10.837 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.834+0000 7fc2027fc640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 0x7fc1fc0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:10.837 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.834+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6398+0+0 (secure 0 0 0) 0x7fc2040ced90 con 0x7fc210101d10 2024-09-16T01:07:10.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.836+0000 7fc215523640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 0x7fc1fc0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:10.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.837+0000 7fc215523640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 0x7fc1fc0a5730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc210101b70 tx=0x7fc20c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:10.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:10.837+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc20409c230 con 0x7fc210101d10 2024-09-16T01:07:11.014 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:11.013+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc21006aea0 con 0x7fc1fc0a3270 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi148 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: osdmap e68: 11 total, 10 up, 11 in 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:07:11.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:07:11.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:11 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi148 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: osdmap e68: 11 total, 10 up, 11 in 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:07:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi148 2024-09-16T01:07:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-16T01:07:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: osdmap e68: 11 total, 10 up, 11 in 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-16T01:07:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:11 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:11.885 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:07:11.422+0000 7fe9f25c4640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T01:07:11.885 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:07:11 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:07:11.436+0000 7fe9ea3e4640 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:07:12.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:12.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:07:12.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: osdmap e69: 11 total, 10 up, 11 in 2024-09-16T01:07:12.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:12 smithi148 ceph-mon[35030]: from='osd.10 ' entity='osd.10' 2024-09-16T01:07:12.455 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:12.455 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:07:12.455 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: osdmap e69: 11 total, 10 up, 11 in 2024-09-16T01:07:12.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:12 smithi193 ceph-mon[35701]: from='osd.10 ' entity='osd.10' 2024-09-16T01:07:12.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:12.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-16T01:07:12.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: osdmap e69: 11 total, 10 up, 11 in 2024-09-16T01:07:12.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:12.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:12 smithi078 ceph-mon[30937]: from='osd.10 ' entity='osd.10' 2024-09-16T01:07:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:07:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:07:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] boot 2024-09-16T01:07:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T01:07:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:13 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:07:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:07:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] boot 2024-09-16T01:07:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T01:07:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:13 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: osd.10 [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] boot 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: osdmap e70: 11 total, 11 up, 11 in 2024-09-16T01:07:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:13 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: osdmap e71: 11 total, 11 up, 11 in 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/3898378965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]': finished 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:14.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:14 smithi193 ceph-mon[35701]: from='client.? 172.21.15.193:0/401454270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:07:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: osdmap e71: 11 total, 11 up, 11 in 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/3898378965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]': finished 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:14.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:14 smithi148 ceph-mon[35030]: from='client.? 172.21.15.193:0/401454270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:07:14.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: osdmap e71: 11 total, 11 up, 11 in 2024-09-16T01:07:14.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/3898378965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]: dispatch 2024-09-16T01:07:14.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9f87f81-392a-4490-9a01-394b494b5251"}]': finished 2024-09-16T01:07:14.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: osdmap e72: 12 total, 11 up, 12 in 2024-09-16T01:07:14.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:14.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:14 smithi078 ceph-mon[30937]: from='client.? 172.21.15.193:0/401454270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-16T01:07:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:15 smithi148 ceph-mon[35030]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:15 smithi148 ceph-mon[35030]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T01:07:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:15 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:15 smithi078 ceph-mon[30937]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:15 smithi078 ceph-mon[30937]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T01:07:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:15 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:15 smithi193 ceph-mon[35701]: pgmap v230: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:15 smithi193 ceph-mon[35701]: osdmap e73: 12 total, 11 up, 12 in 2024-09-16T01:07:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:15 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:16.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:16 smithi078 ceph-mon[30937]: osdmap e74: 12 total, 11 up, 12 in 2024-09-16T01:07:16.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:16.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:16 smithi078 ceph-mon[30937]: pgmap v233: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:16 smithi193 ceph-mon[35701]: osdmap e74: 12 total, 11 up, 12 in 2024-09-16T01:07:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:16 smithi193 ceph-mon[35701]: pgmap v233: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:16 smithi148 ceph-mon[35030]: osdmap e74: 12 total, 11 up, 12 in 2024-09-16T01:07:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:16 smithi148 ceph-mon[35030]: pgmap v233: 1 pgs: 1 remapped+peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:19 smithi193 ceph-mon[35701]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-16T01:07:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:19 smithi148 ceph-mon[35030]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-16T01:07:19.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:19 smithi078 ceph-mon[30937]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-16T01:07:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:21 smithi193 ceph-mon[35701]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T01:07:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:21 smithi148 ceph-mon[35030]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T01:07:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:21 smithi078 ceph-mon[30937]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-16T01:07:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:23 smithi193 ceph-mon[35701]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T01:07:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T01:07:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:23 smithi193 ceph-mon[35701]: Deploying daemon osd.11 on smithi193 2024-09-16T01:07:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:23 smithi148 ceph-mon[35030]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T01:07:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T01:07:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:23.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:23 smithi148 ceph-mon[35030]: Deploying daemon osd.11 on smithi193 2024-09-16T01:07:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:23 smithi078 ceph-mon[30937]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-16T01:07:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-16T01:07:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:23 smithi078 ceph-mon[30937]: Deploying daemon osd.11 on smithi193 2024-09-16T01:07:25.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:25 smithi193 ceph-mon[35701]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T01:07:25.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:25 smithi148 ceph-mon[35030]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T01:07:25.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:25 smithi078 ceph-mon[30937]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-16T01:07:27.272 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:27 smithi193 ceph-mon[35701]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T01:07:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:27 smithi148 ceph-mon[35030]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T01:07:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:27 smithi078 ceph-mon[30937]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-16T01:07:28.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:28.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:28.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:29.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:29 smithi193 ceph-mon[35701]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:07:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:29 smithi148 ceph-mon[35030]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:07:29.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:29 smithi078 ceph-mon[30937]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-16T01:07:31.228 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:31.228 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.228 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.229 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:31.229 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:31.229 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:31 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:31.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:31.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:31 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:31.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:31.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:31 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:31.661 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.660+0000 7fc2027fc640 1 -- 172.21.15.193:0/964726502 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc21006aea0 con 0x7fc1fc0a3270 2024-09-16T01:07:31.661 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-09-16T01:07:31.663 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 msgr2=0x7fc1fc0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 0x7fc1fc0a5730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc210101b70 tx=0x7fc20c009110 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 msgr2=0x7fc210198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc20400d950 tx=0x7fc20400dd50 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 shutdown_connections 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc1fc0a3270 0x7fc1fc0a5730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2101026d0 0x7fc21019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc210101d10 0x7fc210198430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 --2- 172.21.15.193:0/964726502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc210100b10 0x7fc210197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 >> 172.21.15.193:0/964726502 conn(0x7fc2100fc2a0 msgr2=0x7fc21006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 shutdown_connections 2024-09-16T01:07:31.664 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:31.662+0000 7fc217787640 1 -- 172.21.15.193:0/964726502 wait complete. 2024-09-16T01:07:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:32 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:32.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:32 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.444 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.11.service 2024-09-16T01:07:32.446 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-16T01:07:32.447 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T01:07:32.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:32.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:32 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:33.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:33 smithi078 ceph-mon[30937]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:33.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 -- 172.21.15.78:0/1012061863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c102690 msgr2=0x7fcc9c108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1012061863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c102690 0x7fcc9c108f20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009ca0 tx=0x7fcc9402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 -- 172.21.15.78:0/1012061863 shutdown_connections 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1012061863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c102690 0x7fcc9c108f20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1012061863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1012061863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.174+0000 7fcca0e52640 1 -- 172.21.15.78:0/1012061863 >> 172.21.15.78:0/1012061863 conn(0x7fcc9c0fc280 msgr2=0x7fcc9c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:33.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.175+0000 7fcca0e52640 1 -- 172.21.15.78:0/1012061863 shutdown_connections 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.175+0000 7fcca0e52640 1 -- 172.21.15.78:0/1012061863 wait complete. 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.175+0000 7fcca0e52640 1 Processor -- start 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.175+0000 7fcca0e52640 1 -- start start 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 0x7fcc9c1035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c10c750 con 0x7fcc9c101cd0 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c10c5d0 con 0x7fcc9c100ad0 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcca0e52640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c10c8d0 con 0x7fcc9c102690 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45738/0 (socket says 172.21.15.78:45738) 2024-09-16T01:07:33.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 -- 172.21.15.78:0/1207336640 learned_addr learned my addr 172.21.15.78:0/1207336640 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc9ad9d640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 0x7fcc9c1035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc9a59c640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 -- 172.21.15.78:0/1207336640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 msgr2=0x7fcc9c1035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 0x7fcc9c1035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 -- 172.21.15.78:0/1207336640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 msgr2=0x7fcc9c104a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c104a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 -- 172.21.15.78:0/1207336640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c103eb0 con 0x7fcc9c101cd0 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc9ad9d640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 0x7fcc9c1035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc9a59c640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc99d9b640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000e9f0 tx=0x7fcc9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.176+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9000cd80 con 0x7fcc9c101cd0 2024-09-16T01:07:33.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.177+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc90010910 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.177+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c1a20b0 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.177+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc90018990 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.177+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c1a2610 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.178+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9c1a2840 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.179+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fcc90010ac0 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.179+0000 7fcc8b7fe640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 0x7fcc840a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.180+0000 7fcc9a59c640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 0x7fcc840a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.180+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6809+0+0 (secure 0 0 0) 0x7fcc900cf700 con 0x7fcc9c101cd0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.180+0000 7fcc9a59c640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 0x7fcc840a5780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c101b30 tx=0x7fcc8c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:33.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.184+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcc9009c960 con 0x7fcc9c101cd0 2024-09-16T01:07:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.356+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fcc9c1a2a10 con 0x7fcc9c101cd0 2024-09-16T01:07:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.356+0000 7fcc8b7fe640 1 -- 172.21.15.78:0/1207336640 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) ==== 74+0+133 (secure 0 0 0) 0x7fcc9009c260 con 0x7fcc9c101cd0 2024-09-16T01:07:33.358 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 msgr2=0x7fcc840a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 0x7fcc840a5780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c101b30 tx=0x7fcc8c005f90 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 msgr2=0x7fcc9c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000e9f0 tx=0x7fcc9000eeb0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 shutdown_connections 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fcc840a32c0 0x7fcc840a5780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c102690 0x7fcc9c1035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc9c101cd0 0x7fcc9c1030b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 --2- 172.21.15.78:0/1207336640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc9c100ad0 0x7fcc9c104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.358+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 >> 172.21.15.78:0/1207336640 conn(0x7fcc9c0fc280 msgr2=0x7fcc9c06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.359+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 shutdown_connections 2024-09-16T01:07:33.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:33.359+0000 7fcca0e52640 1 -- 172.21.15.78:0/1207336640 wait complete. 2024-09-16T01:07:33.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:33 smithi148 ceph-mon[35030]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:33.430 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:33 smithi193 ceph-mon[35701]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:33.718 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:07:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:07:33.428+0000 7f8628970640 -1 osd.11 0 log_to_monitors true 2024-09-16T01:07:34.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:34 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1207336640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:34.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:34 smithi078 ceph-mon[30937]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:34.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:34 smithi078 ceph-mon[30937]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:34.117 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1726448832,"num_in_osds":12,"osd_in_since":1726448833,"num_remapped_pgs":0} 2024-09-16T01:07:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:34 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1207336640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:34 smithi193 ceph-mon[35701]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:34.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:34 smithi193 ceph-mon[35701]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:34.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:34 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1207336640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:34.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:34 smithi148 ceph-mon[35030]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:34.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:34 smithi148 ceph-mon[35030]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-16T01:07:35.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T01:07:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:35.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T01:07:35.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: osdmap e75: 12 total, 11 up, 12 in 2024-09-16T01:07:35.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:35.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-mon[35701]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.276 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:07:35.068+0000 7f86248f3640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T01:07:35.276 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:07:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:07:35.084+0000 7f862071b640 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:07:35.375 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: osdmap e75: 12 total, 11 up, 12 in 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:35 smithi148 ceph-mon[35030]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: from='osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: osdmap e75: 12 total, 11 up, 12 in 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:35.430 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:35 smithi078 ceph-mon[30937]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- 172.21.15.78:0/2442731655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a68101c00 msgr2=0x7f3a68102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 --2- 172.21.15.78:0/2442731655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a68101c00 0x7f3a68102000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009ca0 tx=0x7f3a5802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- 172.21.15.78:0/2442731655 shutdown_connections 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 --2- 172.21.15.78:0/2442731655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a681037c0 0x7f3a6810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 --2- 172.21.15.78:0/2442731655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68102e00 0x7f3a68103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 --2- 172.21.15.78:0/2442731655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a68101c00 0x7f3a68102000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- 172.21.15.78:0/2442731655 >> 172.21.15.78:0/2442731655 conn(0x7f3a680fd390 msgr2=0x7f3a680ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- 172.21.15.78:0/2442731655 shutdown_connections 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- 172.21.15.78:0/2442731655 wait complete. 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 Processor -- start 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.830+0000 7f3a6da64640 1 -- start start 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68101c00 0x7f3a68197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 0x7f3a68198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810d8e0 con 0x7f3a681037c0 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810d760 con 0x7f3a68102e00 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810da60 con 0x7f3a68101c00 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40018/0 (socket says 172.21.15.78:40018) 2024-09-16T01:07:35.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 -- 172.21.15.78:0/1505538736 learned_addr learned my addr 172.21.15.78:0/1505538736 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a677fe640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68101c00 0x7f3a68197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a66ffd640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 0x7f3a68198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 -- 172.21.15.78:0/1505538736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68101c00 msgr2=0x7f3a68197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68101c00 0x7f3a68197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 -- 172.21.15.78:0/1505538736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 msgr2=0x7f3a68198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 0x7f3a68198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 -- 172.21.15.78:0/1505538736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6819fb50 con 0x7f3a681037c0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a66ffd640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 0x7f3a68198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a67fff640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000c970 tx=0x7f3a6000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60007bb0 con 0x7f3a681037c0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6819fe40 con 0x7f3a681037c0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a60007d50 con 0x7f3a681037c0 2024-09-16T01:07:35.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.831+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60018800 con 0x7f3a681037c0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.832+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a681a0370 con 0x7f3a681037c0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.832+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a68108fa0 con 0x7f3a681037c0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.834+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3a6000fa50 con 0x7f3a681037c0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.834+0000 7f3a64ff9640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 0x7f3a500a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.834+0000 7f3a677fe640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 0x7f3a500a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.834+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6847+0+0 (secure 0 0 0) 0x7f3a600cf760 con 0x7f3a681037c0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.835+0000 7f3a677fe640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 0x7f3a500a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a58006010 tx=0x7f3a58000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:35.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:35.837+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3a60014030 con 0x7f3a681037c0 2024-09-16T01:07:36.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.004+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3a68062df0 con 0x7f3a681037c0 2024-09-16T01:07:36.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.005+0000 7f3a64ff9640 1 -- 172.21.15.78:0/1505538736 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) ==== 74+0+133 (secure 0 0 0) 0x7f3a6009ca40 con 0x7f3a681037c0 2024-09-16T01:07:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.007+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 msgr2=0x7f3a500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.007+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 0x7f3a500a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a58006010 tx=0x7f3a58000f40 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.007+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 msgr2=0x7f3a6819f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.007+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000c970 tx=0x7f3a6000ce30 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 shutdown_connections 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3a500a3270 0x7f3a500a5730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a681037c0 0x7f3a6819f4e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a68102e00 0x7f3a68198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 --2- 172.21.15.78:0/1505538736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a68101c00 0x7f3a68197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 >> 172.21.15.78:0/1505538736 conn(0x7f3a680fd390 msgr2=0x7f3a68106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:36.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 shutdown_connections 2024-09-16T01:07:36.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:36.008+0000 7f3a6da64640 1 -- 172.21.15.78:0/1505538736 wait complete. 2024-09-16T01:07:36.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: osdmap e76: 12 total, 11 up, 12 in 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:36.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:36.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:36 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1505538736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:36.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:07:36.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: osdmap e76: 12 total, 11 up, 12 in 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:36.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:36.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:36 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1505538736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: osdmap e76: 12 total, 11 up, 12 in 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:36.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:36 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1505538736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1726448832,"num_in_osds":12,"osd_in_since":1726448833,"num_remapped_pgs":0} 2024-09-16T01:07:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: purged_snaps scrub starts 2024-09-16T01:07:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: purged_snaps scrub ok 2024-09-16T01:07:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: Detected new or changed devices on smithi193 2024-09-16T01:07:37.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 4272M 2024-09-16T01:07:37.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:37.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] boot 2024-09-16T01:07:37.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T01:07:37.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:37 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:37.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: purged_snaps scrub starts 2024-09-16T01:07:37.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: purged_snaps scrub ok 2024-09-16T01:07:37.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: Detected new or changed devices on smithi193 2024-09-16T01:07:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 4272M 2024-09-16T01:07:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] boot 2024-09-16T01:07:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T01:07:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:37 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: purged_snaps scrub starts 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: purged_snaps scrub ok 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: Detected new or changed devices on smithi193 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 4272M 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-09-16T01:07:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: osd.11 [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] boot 2024-09-16T01:07:37.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: osdmap e77: 12 total, 12 up, 12 in 2024-09-16T01:07:37.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:37 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:07:37.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-16T01:07:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:38.485 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:38 smithi078 ceph-mon[30937]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T01:07:38.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:38 smithi078 ceph-mon[30937]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.562+0000 7fb9e8d40640 1 -- 172.21.15.78:0/1533707634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4102660 msgr2=0x7fb9e4104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.562+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/1533707634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4102660 0x7fb9e4104a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009900 tx=0x7fb9dc02a410 comp rx=0 tx=0).stop 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 -- 172.21.15.78:0/1533707634 shutdown_connections 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/1533707634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e41078c0 0x7fb9e4109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/1533707634 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e4104f90 0x7fb9e4107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/1533707634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4102660 0x7fb9e4104a50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 -- 172.21.15.78:0/1533707634 >> 172.21.15.78:0/1533707634 conn(0x7fb9e40fc280 msgr2=0x7fb9e40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 -- 172.21.15.78:0/1533707634 shutdown_connections 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 -- 172.21.15.78:0/1533707634 wait complete. 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 Processor -- start 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.563+0000 7fb9e8d40640 1 -- start start 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e4102660 0x7fb9e4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e41078c0 0x7fb9e419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410d8e0 con 0x7fb9e4104f90 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410d760 con 0x7fb9e41078c0 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e410da60 con 0x7fb9e4102660 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40036/0 (socket says 172.21.15.78:40036) 2024-09-16T01:07:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 -- 172.21.15.78:0/3596468465 learned_addr learned my addr 172.21.15.78:0/3596468465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e2d9d640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e41078c0 0x7fb9e419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e259c640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e4102660 0x7fb9e4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 -- 172.21.15.78:0/3596468465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e4102660 msgr2=0x7fb9e4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e4102660 0x7fb9e4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 -- 172.21.15.78:0/3596468465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e41078c0 msgr2=0x7fb9e419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e41078c0 0x7fb9e419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 -- 172.21.15.78:0/3596468465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e419fb10 con 0x7fb9e4104f90 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e1d9b640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c970 tx=0x7fb9cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc014070 con 0x7fb9e4104f90 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9cc004440 con 0x7fb9e4104f90 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e419fe00 con 0x7fb9e4104f90 2024-09-16T01:07:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00f450 con 0x7fb9e4104f90 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.564+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e41a0330 con 0x7fb9e4104f90 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.565+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e40fdd70 con 0x7fb9e4104f90 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.567+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb9cc017070 con 0x7fb9e4104f90 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.568+0000 7fb9db7fe640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 0x7fb9d00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.568+0000 7fb9e259c640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 0x7fb9d00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.568+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fb9cc0cf240 con 0x7fb9e4104f90 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.568+0000 7fb9e259c640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 0x7fb9d00a5730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc031610 tx=0x7fb9dc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.572+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb9cc01e020 con 0x7fb9e4104f90 2024-09-16T01:07:38.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:38 smithi148 ceph-mon[35030]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T01:07:38.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:38 smithi148 ceph-mon[35030]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.738+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb9e402da10 con 0x7fb9e4104f90 2024-09-16T01:07:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.738+0000 7fb9db7fe640 1 -- 172.21.15.78:0/3596468465 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) ==== 74+0+133 (secure 0 0 0) 0x7fb9cc09c400 con 0x7fb9e4104f90 2024-09-16T01:07:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 msgr2=0x7fb9d00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 0x7fb9d00a5730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc031610 tx=0x7fb9dc000bc0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 msgr2=0x7fb9e4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c970 tx=0x7fb9cc00ce30 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 shutdown_connections 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb9d00a3270 0x7fb9d00a5730 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9e41078c0 0x7fb9e419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9e4104f90 0x7fb9e4198420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 --2- 172.21.15.78:0/3596468465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9e4102660 0x7fb9e4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 >> 172.21.15.78:0/3596468465 conn(0x7fb9e40fc280 msgr2=0x7fb9e4102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 shutdown_connections 2024-09-16T01:07:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:38.740+0000 7fb9e8d40640 1 -- 172.21.15.78:0/3596468465 wait complete. 2024-09-16T01:07:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:38 smithi193 ceph-mon[35701]: osdmap e78: 12 total, 12 up, 12 in 2024-09-16T01:07:38.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:38 smithi193 ceph-mon[35701]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:39.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:39 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3596468465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:39.429 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1726448856,"num_in_osds":12,"osd_in_since":1726448833,"num_remapped_pgs":0} 2024-09-16T01:07:39.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:07:39.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:39 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3596468465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:39.671 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:39.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:39 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3596468465' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-16T01:07:40.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 -- 172.21.15.78:0/3125620266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854102570 msgr2=0x7f58541029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:40.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3125620266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854102570 0x7f58541029b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5844009900 tx=0x7f584402a480 comp rx=0 tx=0).stop 2024-09-16T01:07:40.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 -- 172.21.15.78:0/3125620266 shutdown_connections 2024-09-16T01:07:40.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3125620266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102ef0 0x7f585410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3125620266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854102570 0x7f58541029b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3125620266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854100e20 0x7f5854101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 -- 172.21.15.78:0/3125620266 >> 172.21.15.78:0/3125620266 conn(0x7f58540fc7d0 msgr2=0x7f58540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 -- 172.21.15.78:0/3125620266 shutdown_connections 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.155+0000 7f585b8e0640 1 -- 172.21.15.78:0/3125620266 wait complete. 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 Processor -- start 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 -- start start 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 0x7f5854198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854102ef0 0x7f585419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5854110fa0 con 0x7f5854100e20 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5854110e20 con 0x7f5854102ef0 2024-09-16T01:07:40.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585b8e0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5854111120 con 0x7f5854102570 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40052/0 (socket says 172.21.15.78:40052) 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 -- 172.21.15.78:0/3519821945 learned_addr learned my addr 172.21.15.78:0/3519821945 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f5858e7b640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 0x7f5854198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f5859e7d640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854102ef0 0x7f585419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 -- 172.21.15.78:0/3519821945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 msgr2=0x7f5854198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 0x7f5854198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 -- 172.21.15.78:0/3519821945 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854102ef0 msgr2=0x7f585419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854102ef0 0x7f585419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 -- 172.21.15.78:0/3519821945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585419d510 con 0x7f5854100e20 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f5858e7b640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 0x7f5854198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:07:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f585967c640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f585000d950 tx=0x7f585000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.156+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850017070 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.157+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5850009c30 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.157+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585419d800 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.157+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585000f6b0 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.157+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58541a50e0 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.158+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5854101200 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.159+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f58500109d0 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.159+0000 7f584a7fc640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 0x7f58400a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.159+0000 7f5858e7b640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 0x7f58400a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.159+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f58500cf1b0 con 0x7f5854100e20 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.160+0000 7f5858e7b640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 0x7f58400a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58540fff50 tx=0x7f5844005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:40.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.163+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5850009de0 con 0x7f5854100e20 2024-09-16T01:07:40.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.329+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58541998c0 con 0x7f5854100e20 2024-09-16T01:07:40.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.330+0000 7f584a7fc640 1 -- 172.21.15.78:0/3519821945 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+18431 (secure 0 0 0) 0x7f585009c370 con 0x7f5854100e20 2024-09-16T01:07:40.332 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:40.333 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:07:37.329633+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.332+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 msgr2=0x7f58400a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.332+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 0x7f58400a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58540fff50 tx=0x7f5844005a20 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 msgr2=0x7f5854198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f585000d950 tx=0x7f585000de10 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 shutdown_connections 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58400a3240 0x7f58400a5700 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5854102ef0 0x7f585419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5854102570 0x7f5854198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 --2- 172.21.15.78:0/3519821945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5854100e20 0x7f5854198560 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 >> 172.21.15.78:0/3519821945 conn(0x7f58540fc7d0 msgr2=0x7f585406a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:40.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 shutdown_connections 2024-09-16T01:07:40.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:40.333+0000 7f585b8e0640 1 -- 172.21.15.78:0/3519821945 wait complete. 2024-09-16T01:07:40.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:40 smithi148 ceph-mon[35030]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:40.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:40 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3519821945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:07:40.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:40 smithi078 ceph-mon[30937]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:40.648 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:40 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3519821945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:07:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:40 smithi193 ceph-mon[35701]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:40 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3519821945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:07:40.960 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:03:52.230028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:07:40.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-16T01:07:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 -- 172.21.15.78:0/2956552209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058102ef0 msgr2=0x7fe05810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 --2- 172.21.15.78:0/2956552209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058102ef0 0x7fe05810da40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe050009ca0 tx=0x7fe05002a980 comp rx=0 tx=0).stop 2024-09-16T01:07:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 -- 172.21.15.78:0/2956552209 shutdown_connections 2024-09-16T01:07:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 --2- 172.21.15.78:0/2956552209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058102ef0 0x7fe05810da40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 --2- 172.21.15.78:0/2956552209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe058102570 0x7fe0581029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 --2- 172.21.15.78:0/2956552209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058100e20 0x7fe058101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 -- 172.21.15.78:0/2956552209 >> 172.21.15.78:0/2956552209 conn(0x7fe0580fc7d0 msgr2=0x7fe0580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 -- 172.21.15.78:0/2956552209 shutdown_connections 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.708+0000 7fe05d729640 1 -- 172.21.15.78:0/2956552209 wait complete. 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 Processor -- start 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 -- start start 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe058102570 0x7fe058199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 0x7fe0581a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe058111060 con 0x7fe058100e20 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe058110ee0 con 0x7fe058102ef0 2024-09-16T01:07:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe05d729640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581111e0 con 0x7fe058102570 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40076/0 (socket says 172.21.15.78:40076) 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 -- 172.21.15.78:0/2959655659 learned_addr learned my addr 172.21.15.78:0/2959655659 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe0577fe640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 0x7fe0581a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 -- 172.21.15.78:0/2959655659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe058102570 msgr2=0x7fe058199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe058102570 0x7fe058199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 -- 172.21.15.78:0/2959655659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 msgr2=0x7fe0581a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 0x7fe0581a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 -- 172.21.15.78:0/2959655659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0581a49d0 con 0x7fe058100e20 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe0577fe640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 0x7fe0581a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.709+0000 7fe056ffd640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe04800b450 tx=0x7fe04800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.710+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048015020 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.710+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe048011d70 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.710+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0581a4ba0 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.710+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04800f340 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.710+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0581a50b0 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.711+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe058101200 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.712+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe048018020 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.713+0000 7fe054ff9640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 0x7fe0440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.713+0000 7fe04ffff640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 0x7fe0440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.713+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fe0480cefe0 con 0x7fe058100e20 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.713+0000 7fe04ffff640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 0x7fe0440a5700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe05806aac0 tx=0x7fe040006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.717+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe04801f020 con 0x7fe058100e20 2024-09-16T01:07:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.885+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe0581a52e0 con 0x7fe058100e20 2024-09-16T01:07:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.885+0000 7fe054ff9640 1 -- 172.21.15.78:0/2959655659 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fe04800d040 con 0x7fe058100e20 2024-09-16T01:07:41.890 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-09-16T01:07:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 msgr2=0x7fe0440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 0x7fe0440a5700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe05806aac0 tx=0x7fe040006c40 comp rx=0 tx=0).stop 2024-09-16T01:07:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 msgr2=0x7fe058199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe04800b450 tx=0x7fe04800b910 comp rx=0 tx=0).stop 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 shutdown_connections 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0440a3240 0x7fe0440a5700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe058102ef0 0x7fe0581a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe058102570 0x7fe058199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 --2- 172.21.15.78:0/2959655659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe058100e20 0x7fe058199410 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 >> 172.21.15.78:0/2959655659 conn(0x7fe0580fc7d0 msgr2=0x7fe058106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 shutdown_connections 2024-09-16T01:07:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:41.887+0000 7fe05d729640 1 -- 172.21.15.78:0/2959655659 wait complete. 2024-09-16T01:07:42.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:41 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2959655659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:07:42.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:41 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2959655659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:07:42.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:41 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2959655659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:07:42.687 INFO:tasks.cephadm:Adding mds.a on smithi078 2024-09-16T01:07:42.687 INFO:tasks.cephadm:Adding mds.d on smithi078 2024-09-16T01:07:42.687 INFO:tasks.cephadm:Adding mds.g on smithi078 2024-09-16T01:07:42.687 INFO:tasks.cephadm:Adding mds.j on smithi078 2024-09-16T01:07:42.687 INFO:tasks.cephadm:Adding mds.b on smithi148 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.e on smithi148 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.h on smithi148 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.k on smithi148 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-09-16T01:07:42.688 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-09-16T01:07:42.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-09-16T01:07:42.947 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:07:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:42 smithi193 ceph-mon[35701]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:42 smithi078 ceph-mon[30937]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:42 smithi148 ceph-mon[35030]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:43.421 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- 172.21.15.193:0/4143320662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 msgr2=0x7f2704102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:43.421 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- 172.21.15.193:0/4143320662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f2704102150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2700009900 tx=0x7f270002a410 comp rx=0 tx=0).stop 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- 172.21.15.193:0/4143320662 shutdown_connections 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- 172.21.15.193:0/4143320662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704102690 0x7f2704108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- 172.21.15.193:0/4143320662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f2704102150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- 172.21.15.193:0/4143320662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704100ad0 0x7f2704100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- 172.21.15.193:0/4143320662 >> 172.21.15.193:0/4143320662 conn(0x7f27040fc280 msgr2=0x7f27040fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- 172.21.15.193:0/4143320662 shutdown_connections 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- 172.21.15.193:0/4143320662 wait complete. 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 Processor -- start 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 -- start start 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704100ad0 0x7f2704193390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.414+0000 7f270bff8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:43.422 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f270bff8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704102690 0x7f270419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f270bff8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f270410d8e0 con 0x7f2704100ad0 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f270bff8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f270410d760 con 0x7f2704102690 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f270bff8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f270410da60 con 0x7f2704101cd0 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:37112/0 (socket says 172.21.15.193:37112) 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 -- 172.21.15.193:0/1600792468 learned_addr learned my addr 172.21.15.193:0/1600792468 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f270a595640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704102690 0x7f270419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709d94640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704100ad0 0x7f2704193390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 -- 172.21.15.193:0/1600792468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704102690 msgr2=0x7f270419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704102690 0x7f270419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.423 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 -- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704100ad0 msgr2=0x7f2704193390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704100ad0 0x7f2704193390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 -- 172.21.15.193:0/1600792468 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270419afc0 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.415+0000 7f2709593640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2700009230 tx=0x7f2700004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.416+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270003f030 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.416+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2700003ce0 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.416+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270419b250 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.416+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700035070 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.416+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27041af250 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.417+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2704105f10 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.419+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2700042030 con 0x7f2704101cd0 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.420+0000 7f26faffd640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 0x7f26f00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:43.424 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.420+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f27000efea0 con 0x7f2704101cd0 2024-09-16T01:07:43.425 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.420+0000 7f2709d94640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 0x7f26f00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:43.425 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.421+0000 7f2709d94640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 0x7f26f00a57b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26f40058c0 tx=0x7f26f400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:43.425 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.423+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27000bca10 con 0x7f2704101cd0 2024-09-16T01:07:43.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.607+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7f27040627f0 con 0x7f26f00a32f0 2024-09-16T01:07:43.616 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.615+0000 7f26faffd640 1 -- 172.21.15.193:0/1600792468 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f27040627f0 con 0x7f26f00a32f0 2024-09-16T01:07:43.616 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-09-16T01:07:43.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 msgr2=0x7f26f00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:43.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 0x7f26f00a57b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26f40058c0 tx=0x7f26f400a2d0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 msgr2=0x7f27041938d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2700009230 tx=0x7f2700004430 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 shutdown_connections 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f26f00a32f0 0x7f26f00a57b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2704102690 0x7f270419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2704101cd0 0x7f27041938d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 --2- 172.21.15.193:0/1600792468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2704100ad0 0x7f2704193390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 >> 172.21.15.193:0/1600792468 conn(0x7f27040fc280 msgr2=0x7f27041030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.617+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 shutdown_connections 2024-09-16T01:07:43.619 INFO:teuthology.orchestra.run.smithi193.stderr:2024-09-16T01:07:43.618+0000 7f270bff8640 1 -- 172.21.15.193:0/1600792468 wait complete. 2024-09-16T01:07:44.314 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.a.service 2024-09-16T01:07:44.317 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.d.service 2024-09-16T01:07:44.360 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.g.service 2024-09-16T01:07:44.402 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.j.service 2024-09-16T01:07:44.446 DEBUG:teuthology.orchestra.run.smithi148:mds.b> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.b.service 2024-09-16T01:07:44.449 DEBUG:teuthology.orchestra.run.smithi148:mds.e> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.e.service 2024-09-16T01:07:44.495 DEBUG:teuthology.orchestra.run.smithi148:mds.h> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.h.service 2024-09-16T01:07:44.539 DEBUG:teuthology.orchestra.run.smithi148:mds.k> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.k.service 2024-09-16T01:07:44.582 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.c.service 2024-09-16T01:07:44.585 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.f.service 2024-09-16T01:07:44.628 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.i.service 2024-09-16T01:07:44.671 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.l.service 2024-09-16T01:07:44.715 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-16T01:07:44.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:07:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:44 smithi148 ceph-mon[35030]: Deploying daemon mds.l on smithi193 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:44.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:44.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:44.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:44.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:44.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:44 smithi193 ceph-mon[35701]: Deploying daemon mds.l on smithi193 2024-09-16T01:07:44.958 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:07:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-09-16T01:07:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:44 smithi078 ceph-mon[30937]: Deploying daemon mds.l on smithi193 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- 172.21.15.78:0/179547227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30106b80 msgr2=0x7fab30106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 --2- 172.21.15.78:0/179547227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30106b80 0x7fab30106f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fab20009ca0 tx=0x7fab2002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- 172.21.15.78:0/179547227 shutdown_connections 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 --2- 172.21.15.78:0/179547227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30100aa0 0x7fab3010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 --2- 172.21.15.78:0/179547227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 --2- 172.21.15.78:0/179547227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30106b80 0x7fab30106f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- 172.21.15.78:0/179547227 >> 172.21.15.78:0/179547227 conn(0x7fab30077230 msgr2=0x7fab300fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- 172.21.15.78:0/179547227 shutdown_connections 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- 172.21.15.78:0/179547227 wait complete. 2024-09-16T01:07:45.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 Processor -- start 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.431+0000 7fab38438640 1 -- start start 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30106b80 0x7fab3019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fab30112010 con 0x7fab30100aa0 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab30111e90 con 0x7fab30100100 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab30112190 con 0x7fab30106b80 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55806/0 (socket says 172.21.15.78:55806) 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 -- 172.21.15.78:0/2068431943 learned_addr learned my addr 172.21.15.78:0/2068431943 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab369d5640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30106b80 0x7fab3019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:45.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab361d4640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 -- 172.21.15.78:0/2068431943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30106b80 msgr2=0x7fab3019ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30106b80 0x7fab3019ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 -- 172.21.15.78:0/2068431943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 msgr2=0x7fab30198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 -- 172.21.15.78:0/2068431943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3019d520 con 0x7fab30100aa0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab361d4640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab359d3640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab2400d950 tx=0x7fab2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24017070 con 0x7fab30100aa0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3019d810 con 0x7fab30100aa0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab24009c30 con 0x7fab30100aa0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.432+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2400f6b0 con 0x7fab30100aa0 2024-09-16T01:07:45.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.433+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab301a50f0 con 0x7fab30100aa0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.433+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab30100560 con 0x7fab30100aa0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.435+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fab240109d0 con 0x7fab30100aa0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.435+0000 7fab2f7fe640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 0x7fab180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.435+0000 7fab361d4640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 0x7fab180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.435+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fab240cf090 con 0x7fab30100aa0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.436+0000 7fab361d4640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 0x7fab180a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab20006010 tx=0x7fab200093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:45.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.439+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab24009de0 con 0x7fab30100aa0 2024-09-16T01:07:45.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.646+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab3010a0f0 con 0x7fab30100aa0 2024-09-16T01:07:45.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.646+0000 7fab2f7fe640 1 -- 172.21.15.78:0/2068431943 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fab2409c250 con 0x7fab30100aa0 2024-09-16T01:07:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:45.648 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"btime":"2024-09-16T01:00:45:346505+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-16T01:07:45.648 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2024-09-16T01:07:45.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 msgr2=0x7fab180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:45.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 0x7fab180a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab20006010 tx=0x7fab200093f0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 msgr2=0x7fab30198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab2400d950 tx=0x7fab2400de10 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 shutdown_connections 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fab180a3240 0x7fab180a5700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab30106b80 0x7fab3019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab30100aa0 0x7fab30198ab0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 --2- 172.21.15.78:0/2068431943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab30100100 0x7fab30198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 >> 172.21.15.78:0/2068431943 conn(0x7fab30077230 msgr2=0x7fab30104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 shutdown_connections 2024-09-16T01:07:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:45.648+0000 7fab38438640 1 -- 172.21.15.78:0/2068431943 wait complete. 2024-09-16T01:07:46.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:45 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2068431943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:46.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:45 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2068431943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:46.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:45 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2068431943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:46.390 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:07:46 smithi193 podman[65614]: 2024-09-16 01:07:46.366750128 +0000 UTC m=+0.340261548 container start 5ac6618a7df0ba967d7d3554473f800141c4aa2cbbc68aaf67737e509cbaf272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-16T01:07:46.390 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mds[65634]: -- 172.21.15.193:0/1537278713 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da7b257860 con 0x55da7b235400 2024-09-16T01:07:46.796 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l[65630]: starting mds.l at 2024-09-16T01:07:46.796 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:07:46 smithi193 bash[65614]: 5ac6618a7df0ba967d7d3554473f800141c4aa2cbbc68aaf67737e509cbaf272 2024-09-16T01:07:46.796 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:07:46 smithi193 systemd[1]: Started Ceph mds.l for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:07:47.202 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6832/4092784262,v1:172.21.15.193:6833/4092784262] up:boot 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: fsmap 1 up:standby 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:47.203 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:46 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6832/4092784262,v1:172.21.15.193:6833/4092784262] up:boot 2024-09-16T01:07:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: fsmap 1 up:standby 2024-09-16T01:07:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:46 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6832/4092784262,v1:172.21.15.193:6833/4092784262] up:boot 2024-09-16T01:07:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: fsmap 1 up:standby 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:46 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:48.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:47 smithi078 ceph-mon[30937]: Deploying daemon mds.k on smithi148 2024-09-16T01:07:48.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:47 smithi193 ceph-mon[35701]: Deploying daemon mds.k on smithi148 2024-09-16T01:07:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:47 smithi148 ceph-mon[35030]: Deploying daemon mds.k on smithi148 2024-09-16T01:07:48.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:07:48.635 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:48 smithi148 systemd[1]: Starting Ceph mds.k for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:07:48.720 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:48.995 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:48 smithi148 podman[63539]: 2024-09-16 01:07:48.606516333 +0000 UTC m=+0.016106138 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:07:48.995 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:48 smithi148 podman[63539]: 2024-09-16 01:07:48.734090291 +0000 UTC m=+0.143680097 container create 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-16T01:07:49.112 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:48 smithi078 ceph-mon[30937]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 -- 172.21.15.78:0/2027663558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 msgr2=0x7ff50410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 --2- 172.21.15.78:0/2027663558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50410da40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009ca0 tx=0x7ff4fc02a950 comp rx=0 tx=0).stop 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 -- 172.21.15.78:0/2027663558 shutdown_connections 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 --2- 172.21.15.78:0/2027663558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50410da40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 --2- 172.21.15.78:0/2027663558 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504102570 0x7ff5041029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 --2- 172.21.15.78:0/2027663558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504100e20 0x7ff504101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.192+0000 7ff50c086640 1 -- 172.21.15.78:0/2027663558 >> 172.21.15.78:0/2027663558 conn(0x7ff5040fc7d0 msgr2=0x7ff5040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.193+0000 7ff50c086640 1 -- 172.21.15.78:0/2027663558 shutdown_connections 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.193+0000 7ff50c086640 1 -- 172.21.15.78:0/2027663558 wait complete. 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.193+0000 7ff50c086640 1 Processor -- start 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.193+0000 7ff50c086640 1 -- start start 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504100e20 0x7ff504197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504102570 0x7ff5041979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff504110f80 con 0x7ff504102ef0 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff504110e00 con 0x7ff504100e20 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff504111100 con 0x7ff504102570 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55830/0 (socket says 172.21.15.78:55830) 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 -- 172.21.15.78:0/1624528839 learned_addr learned my addr 172.21.15.78:0/1624528839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff509621640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504102570 0x7ff5041979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff509e22640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504100e20 0x7ff504197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 -- 172.21.15.78:0/1624528839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504102570 msgr2=0x7ff5041979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504102570 0x7ff5041979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 -- 172.21.15.78:0/1624528839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504100e20 msgr2=0x7ff504197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504100e20 0x7ff504197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 -- 172.21.15.78:0/1624528839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50419c410 con 0x7ff504102ef0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50a623640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc02ae70 tx=0x7ff4fc02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc034040 con 0x7ff504102ef0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4fc030920 con 0x7ff504102ef0 2024-09-16T01:07:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50419c6a0 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.194+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc045480 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.195+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5041a3fe0 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.196+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff504101200 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.197+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff4fc037020 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.197+0000 7ff4faffd640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.198+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7ff4fc0f0480 con 0x7ff504102ef0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.198+0000 7ff509e22640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.198+0000 7ff509e22640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 0x7ff4f00a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff500005fd0 tx=0x7ff50000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.201+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4fc0811b0 con 0x7ff504102ef0 2024-09-16T01:07:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:48 smithi193 ceph-mon[35701]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:48 smithi148 ceph-mon[35030]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:48 smithi148 podman[63539]: 2024-09-16 01:07:48.995390051 +0000 UTC m=+0.404979862 container init 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:48 smithi148 podman[63539]: 2024-09-16 01:07:48.998420747 +0000 UTC m=+0.408010550 container start 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:49 smithi148 ceph-mds[63558]: -- 172.21.15.148:0/586737749 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55773c361860 con 0x55773c33f000 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k[63554]: starting mds.k at 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:49 smithi148 bash[63539]: 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 2024-09-16T01:07:49.386 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:07:49 smithi148 systemd[1]: Started Ceph mds.k for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:07:49.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.409+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff504109090 con 0x7ff504102ef0 2024-09-16T01:07:49.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.409+0000 7ff4faffd640 1 -- 172.21.15.78:0/1624528839 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7ff4fc0bd640 con 0x7ff504102ef0 2024-09-16T01:07:49.410 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:49.411 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"btime":"2024-09-16T01:07:46:396211+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T01:07:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 msgr2=0x7ff4f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 0x7ff4f00a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff500005fd0 tx=0x7ff50000b390 comp rx=0 tx=0).stop 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 msgr2=0x7ff50419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc02ae70 tx=0x7ff4fc02f5d0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 shutdown_connections 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff504102ef0 0x7ff50419bd30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff504102570 0x7ff5041979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 --2- 172.21.15.78:0/1624528839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504100e20 0x7ff504197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 >> 172.21.15.78:0/1624528839 conn(0x7ff5040fc7d0 msgr2=0x7ff5041068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 shutdown_connections 2024-09-16T01:07:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:49.412+0000 7ff50c086640 1 -- 172.21.15.78:0/1624528839 wait complete. 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: Deploying daemon mds.j on smithi078 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1624528839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:50.422 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6832/909591038,v1:172.21.15.148:6833/909591038] up:boot 2024-09-16T01:07:50.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: fsmap 2 up:standby 2024-09-16T01:07:50.423 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:50 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:07:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: Deploying daemon mds.j on smithi078 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1624528839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6832/909591038,v1:172.21.15.148:6833/909591038] up:boot 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: fsmap 2 up:standby 2024-09-16T01:07:50.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:50 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:07:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: Deploying daemon mds.j on smithi078 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1624528839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6832/909591038,v1:172.21.15.148:6833/909591038] up:boot 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: fsmap 2 up:standby 2024-09-16T01:07:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:50 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:07:52.028 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:51 smithi078 systemd[1]: Starting Ceph mds.j for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:07:52.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:07:52.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 podman[70336]: 2024-09-16 01:07:51.981428191 +0000 UTC m=+0.016680004 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:07:52.352 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 podman[70336]: 2024-09-16 01:07:52.106444057 +0000 UTC m=+0.141695869 container create 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-16T01:07:52.559 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:52.683 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 podman[70336]: 2024-09-16 01:07:52.351184632 +0000 UTC m=+0.386436461 container init 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-16T01:07:52.683 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 podman[70336]: 2024-09-16 01:07:52.355972933 +0000 UTC m=+0.391224748 container start 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-16T01:07:52.683 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 ceph-mds[70373]: -- 172.21.15.78:0/2145060208 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560e835d3860 con 0x560e835b0800 2024-09-16T01:07:52.684 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j[70367]: starting mds.j at 2024-09-16T01:07:52.684 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 bash[70336]: 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 2024-09-16T01:07:52.684 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:07:52 smithi078 systemd[1]: Started Ceph mds.j for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 -- 172.21.15.78:0/455419879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102ef0 msgr2=0x7f4fd010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 --2- 172.21.15.78:0/455419879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102ef0 0x7f4fd010da40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009ca0 tx=0x7f4fc802a950 comp rx=0 tx=0).stop 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 -- 172.21.15.78:0/455419879 shutdown_connections 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 --2- 172.21.15.78:0/455419879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102ef0 0x7f4fd010da40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 --2- 172.21.15.78:0/455419879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102570 0x7f4fd01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.089+0000 7f4fd8078640 1 --2- 172.21.15.78:0/455419879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fd0100e20 0x7f4fd0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 -- 172.21.15.78:0/455419879 >> 172.21.15.78:0/455419879 conn(0x7f4fd00fc7d0 msgr2=0x7f4fd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 -- 172.21.15.78:0/455419879 shutdown_connections 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 -- 172.21.15.78:0/455419879 wait complete. 2024-09-16T01:07:53.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 Processor -- start 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 -- start start 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.090+0000 7f4fd8078640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fd0100e20 0x7f4fd0199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102ef0 0x7f4fd01a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0111020 con 0x7f4fd0102570 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0110ea0 con 0x7f4fd0100e20 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd01111a0 con 0x7f4fd0102ef0 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:53.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55872/0 (socket says 172.21.15.78:55872) 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 -- 172.21.15.78:0/3924190365 learned_addr learned my addr 172.21.15.78:0/3924190365 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 -- 172.21.15.78:0/3924190365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102ef0 msgr2=0x7f4fd01a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102ef0 0x7f4fd01a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 -- 172.21.15.78:0/3924190365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fd0100e20 msgr2=0x7f4fd0199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fd0100e20 0x7f4fd0199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 -- 172.21.15.78:0/3924190365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd01a4a10 con 0x7f4fd0102570 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd5613640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400b4a0 tx=0x7f4fc400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4014070 con 0x7f4fd0102570 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fc4010af0 con 0x7f4fd0102570 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4017d10 con 0x7f4fd0102570 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd01a4be0 con 0x7f4fd0102570 2024-09-16T01:07:53.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.091+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd01a5120 con 0x7f4fd0102570 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.092+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd0101280 con 0x7f4fd0102570 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.094+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4fc4017070 con 0x7f4fd0102570 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.094+0000 7f4fc2ffd640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 0x7f4fbc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.094+0000 7f4fd5e14640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 0x7f4fbc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.094+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f4fc40cebf0 con 0x7f4fd0102570 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.095+0000 7f4fd5e14640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 0x7f4fbc0a5780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009710 tx=0x7f4fcc006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:53.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.097+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4fc409bdb0 con 0x7f4fd0102570 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:53.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:53 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:53.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:53 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:53.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.303+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fd006a560 con 0x7f4fd0102570 2024-09-16T01:07:53.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.304+0000 7f4fc2ffd640 1 -- 172.21.15.78:0/3924190365 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3085 (secure 0 0 0) 0x7f4fd006a560 con 0x7f4fd0102570 2024-09-16T01:07:53.305 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:53.305 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"btime":"2024-09-16T01:07:53:034648+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-16T01:07:53.305 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2024-09-16T01:07:53.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 msgr2=0x7f4fbc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:53.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 0x7f4fbc0a5780 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009710 tx=0x7f4fcc006c90 comp rx=0 tx=0).stop 2024-09-16T01:07:53.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 msgr2=0x7f4fd0199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:53.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400b4a0 tx=0x7f4fc400b960 comp rx=0 tx=0).stop 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 shutdown_connections 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f4fbc0a32c0 0x7f4fbc0a5780 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fd0102ef0 0x7f4fd01a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0102570 0x7f4fd0199990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 --2- 172.21.15.78:0/3924190365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fd0100e20 0x7f4fd0199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 >> 172.21.15.78:0/3924190365 conn(0x7f4fd00fc7d0 msgr2=0x7f4fd00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 shutdown_connections 2024-09-16T01:07:53.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:53.306+0000 7f4fd8078640 1 -- 172.21.15.78:0/3924190365 wait complete. 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:53 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:54.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:54 smithi078 ceph-mon[30937]: Deploying daemon mds.d on smithi078 2024-09-16T01:07:54.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:54 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.78:6834/557283089,v1:172.21.15.78:6835/557283089] up:boot 2024-09-16T01:07:54.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:54 smithi078 ceph-mon[30937]: fsmap 3 up:standby 2024-09-16T01:07:54.365 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:54 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:07:54.366 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:54 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3924190365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:54.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:54 smithi148 ceph-mon[35030]: Deploying daemon mds.d on smithi078 2024-09-16T01:07:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:54 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.78:6834/557283089,v1:172.21.15.78:6835/557283089] up:boot 2024-09-16T01:07:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:54 smithi148 ceph-mon[35030]: fsmap 3 up:standby 2024-09-16T01:07:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:54 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:07:54.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:54 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3924190365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:54 smithi193 ceph-mon[35701]: Deploying daemon mds.d on smithi078 2024-09-16T01:07:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:54 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.78:6834/557283089,v1:172.21.15.78:6835/557283089] up:boot 2024-09-16T01:07:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:54 smithi193 ceph-mon[35701]: fsmap 3 up:standby 2024-09-16T01:07:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:54 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:07:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:54 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3924190365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:55.145 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:55 smithi078 ceph-mon[30937]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:55 smithi148 ceph-mon[35030]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:55 smithi193 ceph-mon[35701]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:55.779 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:55 smithi078 systemd[1]: Starting Ceph mds.d for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:07:56.108 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:55 smithi078 podman[70750]: 2024-09-16 01:07:55.733584882 +0000 UTC m=+0.015538244 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:07:56.109 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:55 smithi078 podman[70750]: 2024-09-16 01:07:55.864071048 +0000 UTC m=+0.146024397 container create 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-16T01:07:56.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:07:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:07:56.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 podman[70750]: 2024-09-16 01:07:56.108582075 +0000 UTC m=+0.390535424 container init 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-16T01:07:56.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 podman[70750]: 2024-09-16 01:07:56.111721232 +0000 UTC m=+0.393674584 container start 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:07:56.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 ceph-mds[70769]: -- 172.21.15.78:0/2783639227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5615ed113860 con 0x5615ed0f1400 2024-09-16T01:07:56.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d[70765]: starting mds.d at 2024-09-16T01:07:56.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 bash[70750]: 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 2024-09-16T01:07:56.529 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:07:56 smithi078 systemd[1]: Started Ceph mds.d for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 -- 172.21.15.78:0/2761948875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c100a60 msgr2=0x7f217c100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 --2- 172.21.15.78:0/2761948875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c100a60 0x7f217c100ec0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2164009930 tx=0x7f216402a420 comp rx=0 tx=0).stop 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 -- 172.21.15.78:0/2761948875 shutdown_connections 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 --2- 172.21.15.78:0/2761948875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c101400 0x7f217c10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 --2- 172.21.15.78:0/2761948875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c100a60 0x7f217c100ec0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 --2- 172.21.15.78:0/2761948875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c106a60 0x7f217c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 -- 172.21.15.78:0/2761948875 >> 172.21.15.78:0/2761948875 conn(0x7f217c0fc780 msgr2=0x7f217c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 -- 172.21.15.78:0/2761948875 shutdown_connections 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.001+0000 7f2181957640 1 -- 172.21.15.78:0/2761948875 wait complete. 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 Processor -- start 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 -- start start 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 0x7f217c0fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 0x7f217c0fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f217c110e80 con 0x7f217c101400 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f217c110d00 con 0x7f217c106a60 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f2181957640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f217c111000 con 0x7f217c100a60 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51808/0 (socket says 172.21.15.78:51808) 2024-09-16T01:07:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 -- 172.21.15.78:0/1150497136 learned_addr learned my addr 172.21.15.78:0/1150497136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217b7fe640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 0x7f217c0fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217affd640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 0x7f217c0fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 -- 172.21.15.78:0/1150497136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 msgr2=0x7f217c0fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 0x7f217c0fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 -- 172.21.15.78:0/1150497136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 msgr2=0x7f217c0fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 0x7f217c0fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217a7fc640 1 -- 172.21.15.78:0/1150497136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c1a6960 con 0x7f217c101400 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217affd640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 0x7f217c0fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.002+0000 7f217b7fe640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 0x7f217c0fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f217a7fc640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2164009230 tx=0x7f2164004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216403f030 con 0x7f217c101400 2024-09-16T01:07:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c1a6bf0 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2164003ad0 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164035070 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.003+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c1a7120 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.004+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217c100ec0 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.005+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2164042030 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.006+0000 7f2180955640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 0x7f21680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.006+0000 7f217affd640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 0x7f21680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.006+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f21640efe20 con 0x7f217c101400 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.006+0000 7f217affd640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 0x7f21680a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f216c006fd0 tx=0x7f216c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:07:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.009+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f21640bcfe0 con 0x7f217c101400 2024-09-16T01:07:57.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.215+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f217c02d7c0 con 0x7f217c101400 2024-09-16T01:07:57.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:57.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:57.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:57 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:57.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.215+0000 7f2180955640 1 -- 172.21.15.78:0/1150497136 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7f21640bc8e0 con 0x7f217c101400 2024-09-16T01:07:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:07:57.218 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"btime":"2024-09-16T01:07:57:048240+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-16T01:07:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 5 2024-09-16T01:07:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.218+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 msgr2=0x7f21680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.218+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 0x7f21680a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f216c006fd0 tx=0x7f216c008040 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.218+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 msgr2=0x7f217c0fd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.218+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2164009230 tx=0x7f2164004270 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 shutdown_connections 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f21680a3240 0x7f21680a5700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f217c106a60 0x7f217c0fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f217c101400 0x7f217c0fd3b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 --2- 172.21.15.78:0/1150497136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f217c100a60 0x7f217c0fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 >> 172.21.15.78:0/1150497136 conn(0x7f217c0fc780 msgr2=0x7f217c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 shutdown_connections 2024-09-16T01:07:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:07:57.219+0000 7f2181957640 1 -- 172.21.15.78:0/1150497136 wait complete. 2024-09-16T01:07:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:57 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:57 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mon[35030]: Deploying daemon mds.e on smithi148 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.78:6836/4143414338,v1:172.21.15.78:6837/4143414338] up:boot 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mon[35030]: fsmap 4 up:standby 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1150497136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:58.386 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 systemd[1]: Starting Ceph mds.e for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:07:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:58 smithi078 ceph-mon[30937]: Deploying daemon mds.e on smithi148 2024-09-16T01:07:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:58 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.78:6836/4143414338,v1:172.21.15.78:6837/4143414338] up:boot 2024-09-16T01:07:58.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:58 smithi078 ceph-mon[30937]: fsmap 4 up:standby 2024-09-16T01:07:58.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:58 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:07:58.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:58 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1150497136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:58.545 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:58 smithi193 ceph-mon[35701]: Deploying daemon mds.e on smithi148 2024-09-16T01:07:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:58 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.78:6836/4143414338,v1:172.21.15.78:6837/4143414338] up:boot 2024-09-16T01:07:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:58 smithi193 ceph-mon[35701]: fsmap 4 up:standby 2024-09-16T01:07:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:58 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:07:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:58 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1150497136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:07:58.718 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 podman[63853]: 2024-09-16 01:07:58.345188935 +0000 UTC m=+0.016042314 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:07:58.718 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 podman[63853]: 2024-09-16 01:07:58.47319035 +0000 UTC m=+0.144043718 container create 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-16T01:07:58.718 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 podman[63853]: 2024-09-16 01:07:58.717173227 +0000 UTC m=+0.388026594 container init 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-16T01:07:59.066 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 podman[63853]: 2024-09-16 01:07:58.720252122 +0000 UTC m=+0.391105491 container start 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e) 2024-09-16T01:07:59.067 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-mds[63873]: -- 172.21.15.148:0/3794209436 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f5b6111860 con 0x55f5b60ef000 2024-09-16T01:07:59.067 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: starting mds.e at 2024-09-16T01:07:59.067 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 bash[63853]: 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a 2024-09-16T01:07:59.067 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:07:58 smithi148 systemd[1]: Started Ceph mds.e for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:07:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:07:59 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:59.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:59.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:07:59 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:07:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:07:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:07:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:07:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:07:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:07:59 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:00.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:00.344 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:00 smithi148 ceph-mon[35030]: Deploying daemon mds.f on smithi193 2024-09-16T01:08:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:00 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6834/1399158659,v1:172.21.15.148:6835/1399158659] up:boot 2024-09-16T01:08:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:00 smithi148 ceph-mon[35030]: fsmap 5 up:standby 2024-09-16T01:08:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:00 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:08:00.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:00 smithi193 ceph-mon[35701]: Deploying daemon mds.f on smithi193 2024-09-16T01:08:00.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:00 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6834/1399158659,v1:172.21.15.148:6835/1399158659] up:boot 2024-09-16T01:08:00.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:00 smithi193 ceph-mon[35701]: fsmap 5 up:standby 2024-09-16T01:08:00.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:00 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:08:00.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:00 smithi078 ceph-mon[30937]: Deploying daemon mds.f on smithi193 2024-09-16T01:08:00.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:00 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6834/1399158659,v1:172.21.15.148:6835/1399158659] up:boot 2024-09-16T01:08:00.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:00 smithi078 ceph-mon[30937]: fsmap 5 up:standby 2024-09-16T01:08:00.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:00 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:08:00.744 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:00 smithi193 systemd[1]: Starting Ceph mds.f for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- 172.21.15.78:0/3549691028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 msgr2=0x7f1aec1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 --2- 172.21.15.78:0/3549691028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec1029b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009900 tx=0x7f1ae002a480 comp rx=0 tx=0).stop 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- 172.21.15.78:0/3549691028 shutdown_connections 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 --2- 172.21.15.78:0/3549691028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 0x7f1aec10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 --2- 172.21.15.78:0/3549691028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec1029b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 --2- 172.21.15.78:0/3549691028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 0x7f1aec101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- 172.21.15.78:0/3549691028 >> 172.21.15.78:0/3549691028 conn(0x7f1aec0fc7d0 msgr2=0x7f1aec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- 172.21.15.78:0/3549691028 shutdown_connections 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- 172.21.15.78:0/3549691028 wait complete. 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 Processor -- start 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.826+0000 7f1af3223640 1 -- start start 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 0x7f1aec0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 0x7f1aec0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec110f80 con 0x7f1aec102570 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec110e00 con 0x7f1aec102ef0 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec111100 con 0x7f1aec100e20 2024-09-16T01:08:00.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51814/0 (socket says 172.21.15.78:51814) 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 -- 172.21.15.78:0/1273672919 learned_addr learned my addr 172.21.15.78:0/1273672919 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af0fbf640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 0x7f1aec0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af17c0640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 0x7f1aec0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 -- 172.21.15.78:0/1273672919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 msgr2=0x7f1aec0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 0x7f1aec0fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 -- 172.21.15.78:0/1273672919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 msgr2=0x7f1aec0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 0x7f1aec0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 -- 172.21.15.78:0/1273672919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec1a6910 con 0x7f1aec102570 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1aebfff640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0006010 tx=0x7f1ae0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae003f070 con 0x7f1aec102570 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec1a6ba0 con 0x7f1aec102570 2024-09-16T01:08:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae0031c30 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.827+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0035030 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.828+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec1a70d0 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.829+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec101200 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.831+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1ae0042070 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.831+0000 7f1ae9ffb640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 0x7f1ad40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.831+0000 7f1af0fbf640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 0x7f1ad40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.831+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f1ae00f00d0 con 0x7f1aec102570 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.832+0000 7f1af0fbf640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 0x7f1ad40a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00bbe0 tx=0x7f1adc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:00.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:00.835+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ae0031de0 con 0x7f1aec102570 2024-09-16T01:08:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.037+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aec10a0a0 con 0x7f1aec102570 2024-09-16T01:08:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.037+0000 7f1ae9ffb640 1 -- 172.21.15.78:0/1273672919 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4738 (secure 0 0 0) 0x7f1ae00bd350 con 0x7f1aec102570 2024-09-16T01:08:01.038 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:01.038 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"btime":"2024-09-16T01:07:59:061150+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-16T01:08:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-09-16T01:08:01.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 msgr2=0x7f1ad40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 0x7f1ad40a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00bbe0 tx=0x7f1adc00b390 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 msgr2=0x7f1aec0fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0006010 tx=0x7f1ae0003c10 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 shutdown_connections 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f1ad40a3240 0x7f1ad40a5700 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1aec102ef0 0x7f1aec0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aec102570 0x7f1aec0fd1c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 --2- 172.21.15.78:0/1273672919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100e20 0x7f1aec0fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 >> 172.21.15.78:0/1273672919 conn(0x7f1aec0fc7d0 msgr2=0x7f1aec06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:01.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 shutdown_connections 2024-09-16T01:08:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:01.039+0000 7f1af3223640 1 -- 172.21.15.78:0/1273672919 wait complete. 2024-09-16T01:08:01.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:00 smithi193 podman[65915]: 2024-09-16 01:08:00.866734191 +0000 UTC m=+0.015618308 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:01.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:00 smithi193 podman[65915]: 2024-09-16 01:08:00.971704362 +0000 UTC m=+0.120588470 container create 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-16T01:08:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:01 smithi148 ceph-mon[35030]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:01 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1273672919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:01.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:01 smithi078 ceph-mon[30937]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:01.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:01 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1273672919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:01 smithi193 ceph-mon[35701]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:01 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1273672919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 podman[65915]: 2024-09-16 01:08:01.182621112 +0000 UTC m=+0.331505220 container init 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 podman[65915]: 2024-09-16 01:08:01.185494471 +0000 UTC m=+0.334378581 container start 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 ceph-mds[65934]: -- 172.21.15.193:0/3947932851 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650fd1ef860 con 0x5650fd1cd400 2024-09-16T01:08:01.546 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: starting mds.f at 2024-09-16T01:08:01.547 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 bash[65915]: 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 2024-09-16T01:08:01.547 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:08:01 smithi193 systemd[1]: Started Ceph mds.f for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: Deploying daemon mds.c on smithi193 2024-09-16T01:08:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6834/1850679548,v1:172.21.15.193:6835/1850679548] up:boot 2024-09-16T01:08:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: fsmap 6 up:standby 2024-09-16T01:08:02.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:02 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:08:02.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: Deploying daemon mds.c on smithi193 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6834/1850679548,v1:172.21.15.193:6835/1850679548] up:boot 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: fsmap 6 up:standby 2024-09-16T01:08:02.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:02 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:08:02.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: Deploying daemon mds.c on smithi193 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6834/1850679548,v1:172.21.15.193:6835/1850679548] up:boot 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: fsmap 6 up:standby 2024-09-16T01:08:02.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:02 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:08:03.546 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 systemd[1]: Starting Ceph mds.c for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:03.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:03.848 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 podman[66218]: 2024-09-16 01:08:03.577204118 +0000 UTC m=+0.090145164 container create cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:08:03.848 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 podman[66218]: 2024-09-16 01:08:03.502987593 +0000 UTC m=+0.015928606 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:03.848 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 podman[66218]: 2024-09-16 01:08:03.821578709 +0000 UTC m=+0.334519709 container init cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-16T01:08:03.848 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 podman[66218]: 2024-09-16 01:08:03.824253303 +0000 UTC m=+0.337194303 container start cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T01:08:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:04.296 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 ceph-mds[66238]: -- 172.21.15.193:0/839622836 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562085fad860 con 0x562085f8a800 2024-09-16T01:08:04.438 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: starting mds.c at 2024-09-16T01:08:04.438 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 bash[66218]: cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c 2024-09-16T01:08:04.438 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:08:03 smithi193 systemd[1]: Started Ceph mds.c for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 -- 172.21.15.78:0/2837480821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 msgr2=0x7fadc8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 --2- 172.21.15.78:0/2837480821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8101200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009900 tx=0x7fadbc02a480 comp rx=0 tx=0).stop 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 -- 172.21.15.78:0/2837480821 shutdown_connections 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 --2- 172.21.15.78:0/2837480821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102ef0 0x7fadc810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 --2- 172.21.15.78:0/2837480821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102570 0x7fadc81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 --2- 172.21.15.78:0/2837480821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8101200 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 -- 172.21.15.78:0/2837480821 >> 172.21.15.78:0/2837480821 conn(0x7fadc80fc7d0 msgr2=0x7fadc80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 -- 172.21.15.78:0/2837480821 shutdown_connections 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.596+0000 7fadcefbe640 1 -- 172.21.15.78:0/2837480821 wait complete. 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 Processor -- start 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 -- start start 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102570 0x7fadc8199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 0x7fadc81a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8110f50 con 0x7fadc8100e20 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8110dd0 con 0x7fadc8102ef0 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadcefbe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fadc81110d0 con 0x7fadc8102570 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadccd5a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:04.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadccd5a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48004/0 (socket says 172.21.15.78:48004) 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.597+0000 7fadccd5a640 1 -- 172.21.15.78:0/1772682964 learned_addr learned my addr 172.21.15.78:0/1772682964 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadcd55b640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 0x7fadc81a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadc7fff640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102570 0x7fadc8199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 -- 172.21.15.78:0/1772682964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102570 msgr2=0x7fadc8199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102570 0x7fadc8199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 -- 172.21.15.78:0/1772682964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 msgr2=0x7fadc81a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 0x7fadc81a42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 -- 172.21.15.78:0/1772682964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc81a4800 con 0x7fadc8100e20 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadcd55b640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 0x7fadc81a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadccd5a640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fadbc006010 tx=0x7fadbc003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc03f070 con 0x7fadc8100e20 2024-09-16T01:08:04.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc81a49d0 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadbc031e30 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc035030 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.598+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc81a4e80 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.599+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc8101200 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.601+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fadbc042070 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.602+0000 7fadc5ffb640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 0x7fadb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.602+0000 7fadc7fff640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 0x7fadb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.602+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fadbc0f0480 con 0x7fadc8100e20 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.603+0000 7fadc7fff640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 0x7fadb80a5700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005e10 tx=0x7fadb4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:04.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.605+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fadbc0bd6b0 con 0x7fadc8100e20 2024-09-16T01:08:04.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.808+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadc81a5170 con 0x7fadc8100e20 2024-09-16T01:08:04.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.809+0000 7fadc5ffb640 1 -- 172.21.15.78:0/1772682964 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6394 (secure 0 0 0) 0x7fadbc0bcfb0 con 0x7fadc8100e20 2024-09-16T01:08:04.810 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:04.810 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"btime":"2024-09-16T01:08:03:902906+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-16T01:08:04.810 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 msgr2=0x7fadb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 0x7fadb80a5700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005e10 tx=0x7fadb4005da0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 msgr2=0x7fadc8199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fadbc006010 tx=0x7fadbc003c80 comp rx=0 tx=0).stop 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 shutdown_connections 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fadb80a3240 0x7fadb80a5700 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc8102ef0 0x7fadc81a42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadc8102570 0x7fadc8199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 --2- 172.21.15.78:0/1772682964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadc8100e20 0x7fadc8199240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:04.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 >> 172.21.15.78:0/1772682964 conn(0x7fadc80fc7d0 msgr2=0x7fadc8106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:04.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 shutdown_connections 2024-09-16T01:08:04.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:04.811+0000 7fadcefbe640 1 -- 172.21.15.78:0/1772682964 wait complete. 2024-09-16T01:08:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6836/1637418104,v1:172.21.15.193:6837/1637418104] up:boot 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: fsmap 7 up:standby 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: Deploying daemon mds.h on smithi148 2024-09-16T01:08:05.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:04 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1772682964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:05.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6836/1637418104,v1:172.21.15.193:6837/1637418104] up:boot 2024-09-16T01:08:05.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: fsmap 7 up:standby 2024-09-16T01:08:05.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:08:05.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: Deploying daemon mds.h on smithi148 2024-09-16T01:08:05.138 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1772682964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6836/1637418104,v1:172.21.15.193:6837/1637418104] up:boot 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: fsmap 7 up:standby 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: Deploying daemon mds.h on smithi148 2024-09-16T01:08:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:04 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1772682964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:06.311 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 systemd[1]: Starting Ceph mds.h for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:06.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 podman[64156]: 2024-09-16 01:08:06.213158117 +0000 UTC m=+0.015791932 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:06.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 podman[64156]: 2024-09-16 01:08:06.328099381 +0000 UTC m=+0.130733232 container create f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-16T01:08:07.043 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 podman[64156]: 2024-09-16 01:08:06.658517437 +0000 UTC m=+0.461151244 container init f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:08:07.044 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 podman[64156]: 2024-09-16 01:08:06.661408721 +0000 UTC m=+0.464042532 container start f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T01:08:07.044 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 ceph-mds[64175]: -- 172.21.15.148:0/2775306695 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619c7ca7860 con 0x5619c7c85000 2024-09-16T01:08:07.044 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: starting mds.h at 2024-09-16T01:08:07.044 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 bash[64156]: f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 2024-09-16T01:08:07.044 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:08:06 smithi148 systemd[1]: Started Ceph mds.h for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:07.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:07 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:07 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:07.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:07 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:08.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:08 smithi078 ceph-mon[30937]: Deploying daemon mds.i on smithi193 2024-09-16T01:08:08.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:08 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:boot 2024-09-16T01:08:08.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:08 smithi078 ceph-mon[30937]: fsmap 8 up:standby 2024-09-16T01:08:08.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:08 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 -- 172.21.15.78:0/1268868814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620102ef0 msgr2=0x7fa62010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1268868814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620102ef0 0x7fa62010da40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa618009900 tx=0x7fa61802a480 comp rx=0 tx=0).stop 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 -- 172.21.15.78:0/1268868814 shutdown_connections 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1268868814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620102ef0 0x7fa62010da40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1268868814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102570 0x7fa6201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1268868814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620100e20 0x7fa620101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 -- 172.21.15.78:0/1268868814 >> 172.21.15.78:0/1268868814 conn(0x7fa6200fc7d0 msgr2=0x7fa6200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 -- 172.21.15.78:0/1268868814 shutdown_connections 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.287+0000 7fa625b3d640 1 -- 172.21.15.78:0/1268868814 wait complete. 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.288+0000 7fa625b3d640 1 Processor -- start 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.288+0000 7fa625b3d640 1 -- start start 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620102570 0x7fa6201979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:08.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 0x7fa62019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa620110f80 con 0x7fa620100e20 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa620110e00 con 0x7fa620102ef0 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa620111100 con 0x7fa620102570 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48022/0 (socket says 172.21.15.78:48022) 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1581541040 learned_addr learned my addr 172.21.15.78:0/1581541040 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61effd640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620102570 0x7fa6201979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61ffff640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 0x7fa62019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:08.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1581541040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620102570 msgr2=0x7fa6201979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620102570 0x7fa6201979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1581541040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 msgr2=0x7fa62019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 0x7fa62019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 -- 172.21.15.78:0/1581541040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62019c410 con 0x7fa620100e20 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61ffff640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 0x7fa62019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61f7fe640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa61000ea80 tx=0x7fa61000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61000ce40 con 0x7fa620100e20 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62019c700 con 0x7fa620100e20 2024-09-16T01:08:08.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa610005020 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa610011760 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.289+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6201a3fe0 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.290+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa620101200 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.292+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa6100051d0 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.292+0000 7fa61cff9640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 0x7fa6080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.292+0000 7fa61effd640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 0x7fa6080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.292+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fa6100cfab0 con 0x7fa620100e20 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.293+0000 7fa61effd640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 0x7fa6080a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa6200fece0 tx=0x7fa61400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:08.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.295+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa610016030 con 0x7fa620100e20 2024-09-16T01:08:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:08 smithi148 ceph-mon[35030]: Deploying daemon mds.i on smithi193 2024-09-16T01:08:08.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:08 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:boot 2024-09-16T01:08:08.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:08 smithi148 ceph-mon[35030]: fsmap 8 up:standby 2024-09-16T01:08:08.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:08 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:08:08.485 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:08 smithi193 ceph-mon[35701]: Deploying daemon mds.i on smithi193 2024-09-16T01:08:08.485 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:08 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:boot 2024-09-16T01:08:08.485 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:08 smithi193 ceph-mon[35701]: fsmap 8 up:standby 2024-09-16T01:08:08.485 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:08 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:08:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.496+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6201987c0 con 0x7fa620100e20 2024-09-16T01:08:08.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.497+0000 7fa61cff9640 1 -- 172.21.15.78:0/1581541040 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7219 (secure 0 0 0) 0x7fa6201987c0 con 0x7fa620100e20 2024-09-16T01:08:08.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:08.498 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"btime":"2024-09-16T01:08:07:037066+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-16T01:08:08.498 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2024-09-16T01:08:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 msgr2=0x7fa6080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 0x7fa6080a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa6200fece0 tx=0x7fa61400b390 comp rx=0 tx=0).stop 2024-09-16T01:08:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 msgr2=0x7fa620197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa61000ea80 tx=0x7fa61000ef40 comp rx=0 tx=0).stop 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 shutdown_connections 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa6080a3240 0x7fa6080a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa620102ef0 0x7fa62019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa620102570 0x7fa6201979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 --2- 172.21.15.78:0/1581541040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa620100e20 0x7fa620197460 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 >> 172.21.15.78:0/1581541040 conn(0x7fa6200fc7d0 msgr2=0x7fa6201068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 shutdown_connections 2024-09-16T01:08:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:08.499+0000 7fa625b3d640 1 -- 172.21.15.78:0/1581541040 wait complete. 2024-09-16T01:08:08.796 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:08 smithi193 systemd[1]: Starting Ceph mds.i for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:09.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:09 smithi078 ceph-mon[30937]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:09.163 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:09 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1581541040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:09.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:09 smithi193 ceph-mon[35701]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:09.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:09 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1581541040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:09.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:08 smithi193 podman[66517]: 2024-09-16 01:08:08.782455861 +0000 UTC m=+0.016595324 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:09.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:08 smithi193 podman[66517]: 2024-09-16 01:08:08.900658783 +0000 UTC m=+0.134798237 container create 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T01:08:09.189 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 podman[66517]: 2024-09-16 01:08:09.187389066 +0000 UTC m=+0.421528529 container init 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T01:08:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:09 smithi148 ceph-mon[35030]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:09 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1581541040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:09.549 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 podman[66517]: 2024-09-16 01:08:09.190799738 +0000 UTC m=+0.424939198 container start 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T01:08:09.549 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 ceph-mds[66536]: -- 172.21.15.193:0/395913964 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56184cf6f860 con 0x56184cf4d400 2024-09-16T01:08:09.550 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: starting mds.i at 2024-09-16T01:08:09.550 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 bash[66517]: 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 2024-09-16T01:08:09.550 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:08:09 smithi193 systemd[1]: Started Ceph mds.i for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: Deploying daemon mds.g on smithi078 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:boot 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: fsmap 9 up:standby 2024-09-16T01:08:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:10 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: Deploying daemon mds.g on smithi078 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:boot 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: fsmap 9 up:standby 2024-09-16T01:08:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:10 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: Deploying daemon mds.g on smithi078 2024-09-16T01:08:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:boot 2024-09-16T01:08:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: fsmap 9 up:standby 2024-09-16T01:08:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:10 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:08:11.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:11.491 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 systemd[1]: Starting Ceph mds.g for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:11.752 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 podman[71537]: 2024-09-16 01:08:11.393856997 +0000 UTC m=+0.016370428 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:11.752 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 podman[71537]: 2024-09-16 01:08:11.515854398 +0000 UTC m=+0.138367865 container create 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:08:11.752 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 podman[71537]: 2024-09-16 01:08:11.752192785 +0000 UTC m=+0.374706209 container init 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:08:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:12.028 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 podman[71537]: 2024-09-16 01:08:11.755134929 +0000 UTC m=+0.377648354 container start 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-16T01:08:12.028 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 ceph-mds[71568]: -- 172.21.15.78:0/3461110535 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560bb37f5860 con 0x560bb37d3400 2024-09-16T01:08:12.029 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g[71564]: starting mds.g at 2024-09-16T01:08:12.029 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 bash[71537]: 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e 2024-09-16T01:08:12.029 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:08:11 smithi078 systemd[1]: Started Ceph mds.g for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 -- 172.21.15.78:0/1635757411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 msgr2=0x7f18e4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 --2- 172.21.15.78:0/1635757411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e4100ee0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009930 tx=0x7f18e002a420 comp rx=0 tx=0).stop 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 -- 172.21.15.78:0/1635757411 shutdown_connections 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 --2- 172.21.15.78:0/1635757411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 0x7f18e410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 --2- 172.21.15.78:0/1635757411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e4100ee0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 --2- 172.21.15.78:0/1635757411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 0x7f18e4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 -- 172.21.15.78:0/1635757411 >> 172.21.15.78:0/1635757411 conn(0x7f18e40fc7a0 msgr2=0x7f18e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 -- 172.21.15.78:0/1635757411 shutdown_connections 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.436+0000 7f18ec562640 1 -- 172.21.15.78:0/1635757411 wait complete. 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 Processor -- start 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 -- start start 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 0x7f18e410e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 0x7f18e410ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18e4111fa0 con 0x7f18e4100a80 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18e4111e20 con 0x7f18e4106a80 2024-09-16T01:08:12.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ec562640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18e4112120 con 0x7f18e4101420 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ea2fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ea2fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48068/0 (socket says 172.21.15.78:48068) 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.437+0000 7f18ea2fe640 1 -- 172.21.15.78:0/19280492 learned_addr learned my addr 172.21.15.78:0/19280492 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18eaaff640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 0x7f18e410ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18e9afd640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 0x7f18e410e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 -- 172.21.15.78:0/19280492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 msgr2=0x7f18e410e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 0x7f18e410e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 -- 172.21.15.78:0/19280492 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 msgr2=0x7f18e410ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 0x7f18e410ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 -- 172.21.15.78:0/19280492 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e41a68f0 con 0x7f18e4100a80 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ea2fe640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f18d400c890 tx=0x7f18d400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:12.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4004220 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18d40043c0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e41a6be0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d40049e0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.438+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e41a70f0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.439+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e4100ee0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.440+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f18d40079b0 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.441+0000 7f18db7fe640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 0x7f18d00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.441+0000 7f18e9afd640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 0x7f18d00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.441+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f18d40cf540 con 0x7f18e4100a80 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.441+0000 7f18e9afd640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 0x7f18d00a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000c00 tx=0x7f18e0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:12.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.445+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18d4014030 con 0x7f18e4100a80 2024-09-16T01:08:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.655+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18e4108a60 con 0x7f18e4100a80 2024-09-16T01:08:12.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.656+0000 7f18db7fe640 1 -- 172.21.15.78:0/19280492 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8875 (secure 0 0 0) 0x7f18d409c700 con 0x7f18e4100a80 2024-09-16T01:08:12.657 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:12.657 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"btime":"2024-09-16T01:08:11:833259+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-16T01:08:12.657 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 msgr2=0x7f18d00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 0x7f18d00a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000c00 tx=0x7f18e0031490 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 msgr2=0x7f18e410e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f18d400c890 tx=0x7f18d400cd50 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 shutdown_connections 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18d00a3240 0x7f18d00a5700 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18e4106a80 0x7f18e410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18e4101420 0x7f18e410e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 --2- 172.21.15.78:0/19280492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e4100a80 0x7f18e410e3e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:12.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 >> 172.21.15.78:0/19280492 conn(0x7f18e40fc7a0 msgr2=0x7f18e40fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:12.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 shutdown_connections 2024-09-16T01:08:12.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:12.658+0000 7f18ec562640 1 -- 172.21.15.78:0/19280492 wait complete. 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.78:6838/2440325522,v1:172.21.15.78:6839/2440325522] up:boot 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: fsmap 10 up:standby 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:12.952 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:12.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:12.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:12.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: Deploying daemon mds.a on smithi078 2024-09-16T01:08:12.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:12.953 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:12 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/19280492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:13.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.78:6838/2440325522,v1:172.21.15.78:6839/2440325522] up:boot 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: fsmap 10 up:standby 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: Deploying daemon mds.a on smithi078 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:12 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/19280492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.78:6838/2440325522,v1:172.21.15.78:6839/2440325522] up:boot 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: fsmap 10 up:standby 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: Deploying daemon mds.a on smithi078 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:13.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:12 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/19280492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:14.480 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:14 smithi078 systemd[1]: Starting Ceph mds.a for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:14.755 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:14 smithi078 podman[71935]: 2024-09-16 01:08:14.584339475 +0000 UTC m=+0.015764729 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:14.755 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:14 smithi078 podman[71935]: 2024-09-16 01:08:14.755277387 +0000 UTC m=+0.186702628 container create 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-16T01:08:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:15 smithi078 ceph-mon[30937]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 podman[71935]: 2024-09-16 01:08:15.008059933 +0000 UTC m=+0.439485183 container init 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 podman[71935]: 2024-09-16 01:08:15.011655082 +0000 UTC m=+0.443080324 container start 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 ceph-mds[71954]: -- 172.21.15.78:0/2507936857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7aae67860 con 0x55d7aae44800 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a[71950]: starting mds.a at 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 bash[71935]: 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb 2024-09-16T01:08:15.279 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:08:15 smithi078 systemd[1]: Started Ceph mds.a for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:15 smithi193 ceph-mon[35701]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:15 smithi148 ceph-mon[35030]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:16.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:16.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: Deploying daemon mds.b on smithi148 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.78:6840/643835923,v1:172.21.15.78:6841/643835923] up:boot 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: fsmap 11 up:standby 2024-09-16T01:08:16.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:16 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:16.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: Deploying daemon mds.b on smithi148 2024-09-16T01:08:16.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:16.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.78:6840/643835923,v1:172.21.15.78:6841/643835923] up:boot 2024-09-16T01:08:16.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: fsmap 11 up:standby 2024-09-16T01:08:16.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:16 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: Deploying daemon mds.b on smithi148 2024-09-16T01:08:16.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:16.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.78:6840/643835923,v1:172.21.15.78:6841/643835923] up:boot 2024-09-16T01:08:16.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: fsmap 11 up:standby 2024-09-16T01:08:16.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:16 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:08:16.779 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 -- 172.21.15.78:0/509170779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf281002f0 msgr2=0x7fdf281006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/509170779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf281002f0 0x7fdf281006d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009930 tx=0x7fdf1c02a420 comp rx=0 tx=0).stop 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 -- 172.21.15.78:0/509170779 shutdown_connections 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/509170779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf2810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/509170779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf28100ca0 0x7fdf28106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/509170779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf281002f0 0x7fdf281006d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 -- 172.21.15.78:0/509170779 >> 172.21.15.78:0/509170779 conn(0x7fdf28077210 msgr2=0x7fdf280fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 -- 172.21.15.78:0/509170779 shutdown_connections 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 -- 172.21.15.78:0/509170779 wait complete. 2024-09-16T01:08:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.309+0000 7fdf2ed26640 1 Processor -- start 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 -- start start 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf281002f0 0x7fdf28101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf28101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28112000 con 0x7fdf28100ca0 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28111e80 con 0x7fdf28106f50 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf28112180 con 0x7fdf281002f0 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45082/0 (socket says 172.21.15.78:45082) 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 -- 172.21.15.78:0/2067034266 learned_addr learned my addr 172.21.15.78:0/2067034266 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2d2c3640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf28101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2cac2640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf281002f0 0x7fdf28101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 -- 172.21.15.78:0/2067034266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf281002f0 msgr2=0x7fdf28101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf281002f0 0x7fdf28101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 -- 172.21.15.78:0/2067034266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 msgr2=0x7fdf28101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf28101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 -- 172.21.15.78:0/2067034266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf281a6a20 con 0x7fdf28100ca0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2d2c3640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf28101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf27fff640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400b450 tx=0x7fdf1400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14015020 con 0x7fdf28100ca0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf281a6d10 con 0x7fdf28100ca0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.310+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf14011d70 con 0x7fdf28100ca0 2024-09-16T01:08:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.311+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400f340 con 0x7fdf28100ca0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.311+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf281a71f0 con 0x7fdf28100ca0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.312+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf281006d0 con 0x7fdf28100ca0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.313+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdf14018020 con 0x7fdf28100ca0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.313+0000 7fdf25ffb640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 0x7fdf180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.314+0000 7fdf2cac2640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 0x7fdf180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.314+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fdf140cf000 con 0x7fdf28100ca0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.314+0000 7fdf2cac2640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 0x7fdf180a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.318+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdf1401f020 con 0x7fdf28100ca0 2024-09-16T01:08:17.386 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 systemd[1]: Starting Ceph mds.b for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:08:17.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.517+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf2810a140 con 0x7fdf28100ca0 2024-09-16T01:08:17.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.518+0000 7fdf25ffb640 1 -- 172.21.15.78:0/2067034266 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9698 (secure 0 0 0) 0x7fdf1400d040 con 0x7fdf28100ca0 2024-09-16T01:08:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:17.519 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"btime":"2024-09-16T01:08:16:039693+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-16T01:08:17.519 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2024-09-16T01:08:17.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 msgr2=0x7fdf180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 0x7fdf180a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c005a20 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 msgr2=0x7fdf281017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400b450 tx=0x7fdf1400b910 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 shutdown_connections 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdf180a3240 0x7fdf180a5700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf28106f50 0x7fdf28101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf28100ca0 0x7fdf281017b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 --2- 172.21.15.78:0/2067034266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf281002f0 0x7fdf28101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:17.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.520+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 >> 172.21.15.78:0/2067034266 conn(0x7fdf28077210 msgr2=0x7fdf28105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:17.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.521+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 shutdown_connections 2024-09-16T01:08:17.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:17.521+0000 7fdf2ed26640 1 -- 172.21.15.78:0/2067034266 wait complete. 2024-09-16T01:08:17.744 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:17 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2067034266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:17.745 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 podman[64458]: 2024-09-16 01:08:17.351287165 +0000 UTC m=+0.016059263 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:08:17.745 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 podman[64458]: 2024-09-16 01:08:17.467701029 +0000 UTC m=+0.132473107 container create 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:08:17.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:17 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2067034266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:18.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:17 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2067034266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:18.135 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 podman[64458]: 2024-09-16 01:08:17.744176111 +0000 UTC m=+0.408948190 container init 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:08:18.136 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 podman[64458]: 2024-09-16 01:08:17.747258978 +0000 UTC m=+0.412031060 container start 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-16T01:08:18.136 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 ceph-mds[64477]: -- 172.21.15.148:0/2913920946 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561769aa1860 con 0x561769a7f400 2024-09-16T01:08:18.136 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: starting mds.b at 2024-09-16T01:08:18.136 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 bash[64458]: 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 2024-09-16T01:08:18.136 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:17 smithi148 systemd[1]: Started Ceph mds.b for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:08:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:boot 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: fsmap 12 up:standby 2024-09-16T01:08:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:18 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:19.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:19.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:boot 2024-09-16T01:08:19.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: fsmap 12 up:standby 2024-09-16T01:08:19.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:18 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:08:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:boot 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: fsmap 12 up:standby 2024-09-16T01:08:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:18 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:08:20.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:20.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:20.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:20 smithi078 ceph-mon[30937]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:20.729 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.729 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.729 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:20.730 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:20 smithi148 ceph-mon[35030]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:20.788 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.788 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.788 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.788 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.788 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.789 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.789 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:20.789 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:20.789 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:20.789 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:08:20.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:20 smithi193 ceph-mon[35701]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:21.137 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 -- 172.21.15.78:0/3956848458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 msgr2=0x7fe73c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 --2- 172.21.15.78:0/3956848458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c104e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe730009900 tx=0x7fe73002a410 comp rx=0 tx=0).stop 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 -- 172.21.15.78:0/3956848458 shutdown_connections 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 --2- 172.21.15.78:0/3956848458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c1055b0 0x7fe73c10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 --2- 172.21.15.78:0/3956848458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c104e80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 --2- 172.21.15.78:0/3956848458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c100480 0x7fe73c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 -- 172.21.15.78:0/3956848458 >> 172.21.15.78:0/3956848458 conn(0x7fe73c077210 msgr2=0x7fe73c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 -- 172.21.15.78:0/3956848458 shutdown_connections 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.656+0000 7fe741edd640 1 -- 172.21.15.78:0/3956848458 wait complete. 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 Processor -- start 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 -- start start 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c100480 0x7fe73c197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c1055b0 0x7fe73c19bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c110e90 con 0x7fe73c100e30 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c110d10 con 0x7fe73c100480 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe741edd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c111010 con 0x7fe73c1055b0 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe73affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe73affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45106/0 (socket says 172.21.15.78:45106) 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe73affd640 1 -- 172.21.15.78:0/4068032494 learned_addr learned my addr 172.21.15.78:0/4068032494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe73bfff640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c1055b0 0x7fe73c19bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.657+0000 7fe73b7fe640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c100480 0x7fe73c197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 -- 172.21.15.78:0/4068032494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c1055b0 msgr2=0x7fe73c19bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c1055b0 0x7fe73c19bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 -- 172.21.15.78:0/4068032494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c100480 msgr2=0x7fe73c197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c100480 0x7fe73c197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 -- 172.21.15.78:0/4068032494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73c19c300 con 0x7fe73c100e30 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe73affd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe730006010 tx=0x7fe7300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73003f030 con 0x7fe73c100e30 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73c19c590 con 0x7fe73c100e30 2024-09-16T01:08:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe730003ce0 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730035070 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.658+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73c1a3ed0 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.659+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73c100860 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.660+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe730042030 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.661+0000 7fe738ff9640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 0x7fe7240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.661+0000 7fe73b7fe640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 0x7fe7240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.661+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fe7300efa80 con 0x7fe73c100e30 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.661+0000 7fe73b7fe640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 0x7fe7240a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe72c005fd0 tx=0x7fe72c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.665+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe7300bc6b0 con 0x7fe73c100e30 2024-09-16T01:08:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.890+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe73c1087f0 con 0x7fe73c100e30 2024-09-16T01:08:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.891+0000 7fe738ff9640 1 -- 172.21.15.78:0/4068032494 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10527 (secure 0 0 0) 0x7fe7300c09b0 con 0x7fe73c100e30 2024-09-16T01:08:21.893 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:21.893 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"btime":"2024-09-16T01:08:18:570305+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-16T01:08:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2024-09-16T01:08:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.893+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 msgr2=0x7fe7240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.893+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 0x7fe7240a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe72c005fd0 tx=0x7fe72c00b390 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 msgr2=0x7fe73c197890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe730006010 tx=0x7fe7300042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 shutdown_connections 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe7240a3240 0x7fe7240a5700 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe73c1055b0 0x7fe73c19bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73c100e30 0x7fe73c197890 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 --2- 172.21.15.78:0/4068032494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe73c100480 0x7fe73c197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 >> 172.21.15.78:0/4068032494 conn(0x7fe73c077210 msgr2=0x7fe73c103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 shutdown_connections 2024-09-16T01:08:21.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:21.894+0000 7fe741edd640 1 -- 172.21.15.78:0/4068032494 wait complete. 2024-09-16T01:08:22.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4068032494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:22.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:22 smithi078 ceph-mon[30937]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:22.616 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4068032494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:22 smithi193 ceph-mon[35701]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:22.870 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:22.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4068032494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:22.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:22 smithi148 ceph-mon[35030]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:23.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 -- 172.21.15.78:0/465749204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102570 msgr2=0x7f66c01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 --2- 172.21.15.78:0/465749204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102570 0x7f66c01029b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009900 tx=0x7f66a802a480 comp rx=0 tx=0).stop 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 -- 172.21.15.78:0/465749204 shutdown_connections 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 --2- 172.21.15.78:0/465749204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102ef0 0x7f66c010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 --2- 172.21.15.78:0/465749204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102570 0x7f66c01029b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 --2- 172.21.15.78:0/465749204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 -- 172.21.15.78:0/465749204 >> 172.21.15.78:0/465749204 conn(0x7f66c00fc7d0 msgr2=0x7f66c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 -- 172.21.15.78:0/465749204 shutdown_connections 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 -- 172.21.15.78:0/465749204 wait complete. 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.360+0000 7f66c538c640 1 Processor -- start 2024-09-16T01:08:23.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 -- start start 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c01972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102570 0x7f66c01977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0110f80 con 0x7f66c0102ef0 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0110e00 con 0x7f66c0100e20 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66c538c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0111100 con 0x7f66c0102570 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45126/0 (socket says 172.21.15.78:45126) 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 -- 172.21.15.78:0/3901695033 learned_addr learned my addr 172.21.15.78:0/3901695033 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66be7fc640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102570 0x7f66c01977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:23.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66beffd640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c01972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 -- 172.21.15.78:0/3901695033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102570 msgr2=0x7f66c01977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102570 0x7f66c01977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 -- 172.21.15.78:0/3901695033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 msgr2=0x7f66c01972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c01972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 -- 172.21.15.78:0/3901695033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c019c260 con 0x7f66c0102ef0 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66beffd640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c01972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66bf7fe640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66b800c970 tx=0x7f66b800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.361+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8007bb0 con 0x7f66c0102ef0 2024-09-16T01:08:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.362+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c019c550 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.362+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b8007d50 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.362+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8018800 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.362+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c0069f70 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.363+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c0101200 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.364+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f66b800fa50 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.364+0000 7f66b7fff640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 0x7f66ac0a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.364+0000 7f66beffd640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 0x7f66ac0a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.364+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7f66b80cf560 con 0x7f66c0102ef0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.365+0000 7f66beffd640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 0x7f66ac0a7910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66b0008e00 tx=0x7f66b0008350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.368+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f66b8014030 con 0x7f66c0102ef0 2024-09-16T01:08:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.535+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f66c006a230 con 0x7f66c0102ef0 2024-09-16T01:08:23.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.536+0000 7f66b7fff640 1 -- 172.21.15.78:0/3901695033 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) ==== 84+0+65 (secure 0 0 0) 0x7f66b809c720 con 0x7f66c0102ef0 2024-09-16T01:08:23.537 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:23.537 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-09-16T01:08:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:08:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-09-16T01:08:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-09-16T01:08:23.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:08:23.538 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-09-16T01:08:23.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.537+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 msgr2=0x7f66ac0a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.537+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 0x7f66ac0a7910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f66b0008e00 tx=0x7f66b0008350 comp rx=0 tx=0).stop 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.537+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 msgr2=0x7f66c019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.537+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66b800c970 tx=0x7f66b800ce30 comp rx=0 tx=0).stop 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 shutdown_connections 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66ac0a5450 0x7f66ac0a7910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66c0102ef0 0x7f66c019bb80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66c0102570 0x7f66c01977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 --2- 172.21.15.78:0/3901695033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66c0100e20 0x7f66c01972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 >> 172.21.15.78:0/3901695033 conn(0x7f66c00fc7d0 msgr2=0x7f66c0106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 shutdown_connections 2024-09-16T01:08:23.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:23.538+0000 7f66c538c640 1 -- 172.21.15.78:0/3901695033 wait complete. 2024-09-16T01:08:24.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:23 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3901695033' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:23 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3901695033' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T01:08:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:08:24.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:08:24.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:08:24.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:23 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3901695033' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-16T01:08:24.181 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:08:24.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:08:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:24.858 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:24 smithi078 ceph-mon[30937]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 -- 172.21.15.78:0/3645716057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 msgr2=0x7fd310106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 --2- 172.21.15.78:0/3645716057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd310106f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd300009900 tx=0x7fd30002a410 comp rx=0 tx=0).stop 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 -- 172.21.15.78:0/3645716057 shutdown_connections 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 --2- 172.21.15.78:0/3645716057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3101005a0 0x7fd31010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 --2- 172.21.15.78:0/3645716057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3100ffc00 0x7fd310100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 --2- 172.21.15.78:0/3645716057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd310106f60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 -- 172.21.15.78:0/3645716057 >> 172.21.15.78:0/3645716057 conn(0x7fd3100fba90 msgr2=0x7fd3100fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:24.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 -- 172.21.15.78:0/3645716057 shutdown_connections 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.940+0000 7fd316858640 1 -- 172.21.15.78:0/3645716057 wait complete. 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 Processor -- start 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 -- start start 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 0x7fd3101983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3101005a0 0x7fd3101988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd310111fa0 con 0x7fd310106b80 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd310111e20 con 0x7fd3100ffc00 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd316858640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd310112120 con 0x7fd3101005a0 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd314df5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd314df5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42674/0 (socket says 172.21.15.78:42674) 2024-09-16T01:08:24.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd314df5640 1 -- 172.21.15.78:0/2196629646 learned_addr learned my addr 172.21.15.78:0/2196629646 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd30f7fe640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3101005a0 0x7fd3101988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.941+0000 7fd30ffff640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 0x7fd3101983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 -- 172.21.15.78:0/2196629646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3101005a0 msgr2=0x7fd3101988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3101005a0 0x7fd3101988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 -- 172.21.15.78:0/2196629646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 msgr2=0x7fd3101983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 0x7fd3101983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 -- 172.21.15.78:0/2196629646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31019d350 con 0x7fd310106b80 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd30ffff640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 0x7fd3101983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd314df5640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd30800ccb0 tx=0x7fd30800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308003970 con 0x7fd310106b80 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd308004030 con 0x7fd310106b80 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30800f720 con 0x7fd310106b80 2024-09-16T01:08:24.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31019d640 con 0x7fd310106b80 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.942+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3101a4f20 con 0x7fd310106b80 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.943+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd310100060 con 0x7fd310106b80 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.944+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd3080041e0 con 0x7fd310106b80 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.945+0000 7fd30d7fa640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 0x7fd2f80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.945+0000 7fd30ffff640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 0x7fd2f80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.945+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7127+0+0 (secure 0 0 0) 0x7fd3080cf4e0 con 0x7fd310106b80 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.945+0000 7fd30ffff640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 0x7fd2f80a5780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd300005ab0 tx=0x7fd300005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:24.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:24.949+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd308014070 con 0x7fd310106b80 2024-09-16T01:08:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:24 smithi193 ceph-mon[35701]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:25.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.118+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd3101a5150 con 0x7fd310106b80 2024-09-16T01:08:25.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:24 smithi148 ceph-mon[35030]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.690+0000 7fd30d7fa640 1 -- 172.21.15.78:0/2196629646 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) ==== 138+0+0 (secure 0 0 0) 0x7fd30809c6a0 con 0x7fd310106b80 2024-09-16T01:08:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 msgr2=0x7fd2f80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 0x7fd2f80a5780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd300005ab0 tx=0x7fd300005a20 comp rx=0 tx=0).stop 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 msgr2=0x7fd31019cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd30800ccb0 tx=0x7fd30800a620 comp rx=0 tx=0).stop 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 shutdown_connections 2024-09-16T01:08:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2f80a32c0 0x7fd2f80a5780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd310106b80 0x7fd31019cc70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3101005a0 0x7fd3101988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 --2- 172.21.15.78:0/2196629646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3100ffc00 0x7fd3101983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 >> 172.21.15.78:0/2196629646 conn(0x7fd3100fba90 msgr2=0x7fd3100fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 shutdown_connections 2024-09-16T01:08:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:25.692+0000 7fd316858640 1 -- 172.21.15.78:0/2196629646 wait complete. 2024-09-16T01:08:26.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:25 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T01:08:26.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:25 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T01:08:26.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:25 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-16T01:08:26.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-16T01:08:26.618 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:26 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T01:08:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:26 smithi078 ceph-mon[30937]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T01:08:27.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:26 smithi078 ceph-mon[30937]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:26 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T01:08:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:26 smithi193 ceph-mon[35701]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T01:08:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:26 smithi193 ceph-mon[35701]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 -- 172.21.15.78:0/3310728321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c106a80 msgr2=0x7f231c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 --2- 172.21.15.78:0/3310728321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c106a80 0x7f231c106e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2318009930 tx=0x7f231802a420 comp rx=0 tx=0).stop 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 -- 172.21.15.78:0/3310728321 shutdown_connections 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 --2- 172.21.15.78:0/3310728321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c101420 0x7f231c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 --2- 172.21.15.78:0/3310728321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c100a80 0x7f231c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 --2- 172.21.15.78:0/3310728321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c106a80 0x7f231c106e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 -- 172.21.15.78:0/3310728321 >> 172.21.15.78:0/3310728321 conn(0x7f231c0fc7a0 msgr2=0x7f231c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 -- 172.21.15.78:0/3310728321 shutdown_connections 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.127+0000 7f2323b84640 1 -- 172.21.15.78:0/3310728321 wait complete. 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 Processor -- start 2024-09-16T01:08:27.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 -- start start 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 0x7f231c197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c106a80 0x7f231c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f231c110e80 con 0x7f231c100a80 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f231c110d00 con 0x7f231c101420 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2323b84640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f231c111000 con 0x7f231c106a80 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42684/0 (socket says 172.21.15.78:42684) 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 -- 172.21.15.78:0/2385751001 learned_addr learned my addr 172.21.15.78:0/2385751001 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f232111f640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 0x7f231c197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:27.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 -- 172.21.15.78:0/2385751001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c106a80 msgr2=0x7f231c19bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2322121640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c106a80 0x7f231c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c106a80 0x7f231c19bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 -- 172.21.15.78:0/2385751001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 msgr2=0x7f231c197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 0x7f231c197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f2321920640 1 -- 172.21.15.78:0/2385751001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c19c370 con 0x7f231c100a80 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.128+0000 7f232111f640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 0x7f231c197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2321920640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2318009230 tx=0x7f2318004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231803f030 con 0x7f231c100a80 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2318003ad0 con 0x7f231c100a80 2024-09-16T01:08:27.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231c19c600 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318035070 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.129+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231c1a3f40 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.130+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f231c1021c0 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.131+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2318042030 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.132+0000 7f2312ffd640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 0x7f23080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.132+0000 7f232111f640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 0x7f23080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.132+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7497+0+0 (secure 0 0 0) 0x7f23180effa0 con 0x7f231c100a80 2024-09-16T01:08:27.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.132+0000 7f232111f640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 0x7f23080a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f231c0fedb0 tx=0x7f230c005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:27.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:26 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2196629646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-16T01:08:27.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:26 smithi148 ceph-mon[35030]: osdmap e79: 12 total, 12 up, 12 in 2024-09-16T01:08:27.139 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:26 smithi148 ceph-mon[35030]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:27.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.136+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f23180bcff0 con 0x7f231c100a80 2024-09-16T01:08:27.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.302+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f231c108ff0 con 0x7f231c100a80 2024-09-16T01:08:27.698 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-09-16T01:08:27.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.697+0000 7f2312ffd640 1 -- 172.21.15.78:0/2385751001 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) ==== 171+0+0 (secure 0 0 0) 0x7f23180bc8f0 con 0x7f231c100a80 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 msgr2=0x7f23080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 0x7f23080a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f231c0fedb0 tx=0x7f230c005e50 comp rx=0 tx=0).stop 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 msgr2=0x7f231c1973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2318009230 tx=0x7f2318004270 comp rx=0 tx=0).stop 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 shutdown_connections 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f23080a3240 0x7f23080a5700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f231c106a80 0x7f231c19bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c101420 0x7f231c197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 --2- 172.21.15.78:0/2385751001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f231c100a80 0x7f231c1973c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:27.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 >> 172.21.15.78:0/2385751001 conn(0x7f231c0fc7a0 msgr2=0x7f231c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:27.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 shutdown_connections 2024-09-16T01:08:27.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:27.699+0000 7f2323b84640 1 -- 172.21.15.78:0/2385751001 wait complete. 2024-09-16T01:08:28.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:27 smithi078 ceph-mon[30937]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:28.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:27 smithi078 ceph-mon[30937]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T01:08:28.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:27 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T01:08:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:27 smithi193 ceph-mon[35701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:27 smithi193 ceph-mon[35701]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T01:08:28.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:27 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T01:08:28.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:27 smithi148 ceph-mon[35030]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:28.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:27 smithi148 ceph-mon[35030]: osdmap e80: 12 total, 12 up, 12 in 2024-09-16T01:08:28.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:27 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-16T01:08:28.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:08:28.711 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:28.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:28 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T01:08:28.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:28 smithi078 ceph-mon[30937]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T01:08:28.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:28 smithi078 ceph-mon[30937]: pgmap v276: 97 pgs: 11 active+clean, 2 creating+activating, 20 creating+peering, 64 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:28.974 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:28 smithi078 ceph-mon[30937]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T01:08:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:28 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T01:08:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:28 smithi193 ceph-mon[35701]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T01:08:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:28 smithi193 ceph-mon[35701]: pgmap v276: 97 pgs: 11 active+clean, 2 creating+activating, 20 creating+peering, 64 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:28 smithi193 ceph-mon[35701]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T01:08:29.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:28 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2385751001' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-16T01:08:29.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:28 smithi148 ceph-mon[35030]: osdmap e81: 12 total, 12 up, 12 in 2024-09-16T01:08:29.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:28 smithi148 ceph-mon[35030]: pgmap v276: 97 pgs: 11 active+clean, 2 creating+activating, 20 creating+peering, 64 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:29.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:28 smithi148 ceph-mon[35030]: osdmap e82: 12 total, 12 up, 12 in 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 -- 172.21.15.78:0/1413332776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c101de0 msgr2=0x7f2a0c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 --2- 172.21.15.78:0/1413332776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c101de0 0x7f2a0c1021c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009900 tx=0x7f2a0802a410 comp rx=0 tx=0).stop 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 -- 172.21.15.78:0/1413332776 shutdown_connections 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 --2- 172.21.15.78:0/1413332776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c106f10 0x7f2a0c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 --2- 172.21.15.78:0/1413332776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c102790 0x7f2a0c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 --2- 172.21.15.78:0/1413332776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c101de0 0x7f2a0c1021c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 -- 172.21.15.78:0/1413332776 >> 172.21.15.78:0/1413332776 conn(0x7f2a0c0fd8c0 msgr2=0x7f2a0c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 -- 172.21.15.78:0/1413332776 shutdown_connections 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.172+0000 7f2a147f6640 1 -- 172.21.15.78:0/1413332776 wait complete. 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 Processor -- start 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 -- start start 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c101de0 0x7f2a0c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 0x7f2a0c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c111fb0 con 0x7f2a0c102790 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c111e30 con 0x7f2a0c106f10 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a147f6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c112130 con 0x7f2a0c101de0 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42702/0 (socket says 172.21.15.78:42702) 2024-09-16T01:08:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 -- 172.21.15.78:0/895476542 learned_addr learned my addr 172.21.15.78:0/895476542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a12592640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c101de0 0x7f2a0c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a12d93640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 0x7f2a0c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 -- 172.21.15.78:0/895476542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c101de0 msgr2=0x7f2a0c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c101de0 0x7f2a0c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 -- 172.21.15.78:0/895476542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 msgr2=0x7f2a0c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 0x7f2a0c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a11d91640 1 -- 172.21.15.78:0/895476542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c19d4c0 con 0x7f2a0c102790 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.173+0000 7f2a12d93640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 0x7f2a0c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a11d91640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b4a0 tx=0x7f29fc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc014070 con 0x7f2a0c102790 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29fc010af0 con 0x7f2a0c102790 2024-09-16T01:08:29.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc017d10 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0c19d7b0 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.174+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a0c1a5090 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.175+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a0c1021c0 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.177+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f29fc017070 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.177+0000 7f2a037fe640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 0x7f29f80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.178+0000 7f2a12592640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 0x7f29f80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.178+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7879+0+0 (secure 0 0 0) 0x7f29fc0cef10 con 0x7f2a0c102790 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.178+0000 7f2a12592640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 0x7f29f80a5780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009230 tx=0x7f2a080058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:29.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.181+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f29fc09bd40 con 0x7f2a0c102790 2024-09-16T01:08:29.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.383+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2a0c10a150 con 0x7f2a0c102790 2024-09-16T01:08:29.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.729+0000 7f2a037fe640 1 -- 172.21.15.78:0/895476542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f29fc097070 con 0x7f2a0c102790 2024-09-16T01:08:29.730 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 msgr2=0x7f29f80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 0x7f29f80a5780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a08009230 tx=0x7f2a080058d0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 msgr2=0x7f2a0c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b4a0 tx=0x7f29fc00b960 comp rx=0 tx=0).stop 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 shutdown_connections 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29f80a32c0 0x7f29f80a5780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a0c106f10 0x7f2a0c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a0c102790 0x7f2a0c198a50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 --2- 172.21.15.78:0/895476542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a0c101de0 0x7f2a0c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:29.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 >> 172.21.15.78:0/895476542 conn(0x7f2a0c0fd8c0 msgr2=0x7f2a0c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:29.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 shutdown_connections 2024-09-16T01:08:29.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:29.731+0000 7f2a147f6640 1 -- 172.21.15.78:0/895476542 wait complete. 2024-09-16T01:08:30.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:29 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T01:08:30.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:29 smithi078 ceph-mon[30937]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T01:08:30.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:29 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T01:08:30.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:29 smithi193 ceph-mon[35701]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T01:08:30.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:29 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-16T01:08:30.127 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:29 smithi148 ceph-mon[35030]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-16T01:08:30.478 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-16T01:08:30.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-16T01:08:30.722 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:30.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T01:08:30.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T01:08:30.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T01:08:30.723 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T01:08:30.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: pgmap v279: 97 pgs: 17 active+clean, 2 creating+activating, 26 creating+peering, 52 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:30.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:active 2024-09-16T01:08:30.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:30 smithi078 ceph-mon[30937]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: pgmap v279: 97 pgs: 17 active+clean, 2 creating+activating, 26 creating+peering, 52 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:active 2024-09-16T01:08:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:30 smithi193 ceph-mon[35701]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:31.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: osdmap e83: 12 total, 12 up, 12 in 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/895476542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: pgmap v279: 97 pgs: 17 active+clean, 2 creating+activating, 26 creating+peering, 52 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] up:active 2024-09-16T01:08:31.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:30 smithi148 ceph-mon[35030]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 -- 172.21.15.78:0/2568734231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa0010c170 msgr2=0x7faa0010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 --2- 172.21.15.78:0/2568734231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa0010c170 0x7faa0010e610 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009900 tx=0x7fa9f802a480 comp rx=0 tx=0).stop 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 -- 172.21.15.78:0/2568734231 shutdown_connections 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 --2- 172.21.15.78:0/2568734231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa0010c170 0x7faa0010e610 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 --2- 172.21.15.78:0/2568734231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa0010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 --2- 172.21.15.78:0/2568734231 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa00102d40 0x7faa00103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 -- 172.21.15.78:0/2568734231 >> 172.21.15.78:0/2568734231 conn(0x7faa000fc7a0 msgr2=0x7faa000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 -- 172.21.15.78:0/2568734231 shutdown_connections 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.234+0000 7faa059a3640 1 -- 172.21.15.78:0/2568734231 wait complete. 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 Processor -- start 2024-09-16T01:08:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 -- start start 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa001989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa0010c170 0x7faa0019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faa00111fd0 con 0x7faa00102d40 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7faa00111e50 con 0x7faa0010c170 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.235+0000 7faa059a3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faa00112150 con 0x7faa00103660 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42726/0 (socket says 172.21.15.78:42726) 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 -- 172.21.15.78:0/3585634981 learned_addr learned my addr 172.21.15.78:0/3585634981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9fe7fc640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa001989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9ff7fe640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa0010c170 0x7faa0019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 -- 172.21.15.78:0/3585634981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 msgr2=0x7faa001989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa001989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 -- 172.21.15.78:0/3585634981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa0010c170 msgr2=0x7faa0019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa0010c170 0x7faa0019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 -- 172.21.15.78:0/3585634981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0019d420 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9fe7fc640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa001989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7fa9feffd640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000ba50 tx=0x7fa9f000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0019070 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0019d710 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9f000cbe0 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00076e0 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.236+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001a4ff0 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.238+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa00103120 con 0x7faa00102d40 2024-09-16T01:08:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.238+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa9f001c070 con 0x7faa00102d40 2024-09-16T01:08:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.239+0000 7faa049a1640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 0x7fa9e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.239+0000 7fa9fe7fc640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 0x7fa9e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.239+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7979+0+0 (secure 0 0 0) 0x7fa9f00cee20 con 0x7faa00102d40 2024-09-16T01:08:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.239+0000 7fa9fe7fc640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 0x7fa9e80a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005fd0 tx=0x7fa9f4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.243+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa9f009bc90 con 0x7faa00102d40 2024-09-16T01:08:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.419+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7faa0002d7c0 con 0x7faa00102d40 2024-09-16T01:08:31.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.731+0000 7faa049a1640 1 -- 172.21.15.78:0/3585634981 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v84) ==== 157+0+0 (secure 0 0 0) 0x7fa9f009b590 con 0x7faa00102d40 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 msgr2=0x7fa9e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 0x7fa9e80a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005fd0 tx=0x7fa9f4005850 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 msgr2=0x7faa00198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000ba50 tx=0x7fa9f000bf10 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 shutdown_connections 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa9e80a3240 0x7fa9e80a5700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faa0010c170 0x7faa0019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa00103660 0x7faa001989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 --2- 172.21.15.78:0/3585634981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa00102d40 0x7faa00198470 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 >> 172.21.15.78:0/3585634981 conn(0x7faa000fc7a0 msgr2=0x7faa000fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:31.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 shutdown_connections 2024-09-16T01:08:31.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:31.734+0000 7faa059a3640 1 -- 172.21.15.78:0/3585634981 wait complete. 2024-09-16T01:08:32.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:31 smithi078 ceph-mon[30937]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T01:08:32.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:31 smithi078 ceph-mon[30937]: Cluster is now healthy 2024-09-16T01:08:32.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:31 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T01:08:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:31 smithi193 ceph-mon[35701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T01:08:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:31 smithi193 ceph-mon[35701]: Cluster is now healthy 2024-09-16T01:08:32.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:31 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T01:08:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:31 smithi148 ceph-mon[35030]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-16T01:08:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:31 smithi148 ceph-mon[35030]: Cluster is now healthy 2024-09-16T01:08:32.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:31 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T01:08:32.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:08:32.704 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:32.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:32 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T01:08:32.988 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:32 smithi078 ceph-mon[30937]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T01:08:32.989 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:32 smithi078 ceph-mon[30937]: pgmap v281: 97 pgs: 62 active+clean, 12 creating+peering, 23 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-16T01:08:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:32 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T01:08:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:32 smithi193 ceph-mon[35701]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T01:08:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:32 smithi193 ceph-mon[35701]: pgmap v281: 97 pgs: 62 active+clean, 12 creating+peering, 23 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-16T01:08:33.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:32 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3585634981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-16T01:08:33.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:32 smithi148 ceph-mon[35030]: osdmap e84: 12 total, 12 up, 12 in 2024-09-16T01:08:33.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:32 smithi148 ceph-mon[35030]: pgmap v281: 97 pgs: 62 active+clean, 12 creating+peering, 23 unknown; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-16T01:08:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 -- 172.21.15.78:0/2630681443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 msgr2=0x7fb7b41063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 --2- 172.21.15.78:0/2630681443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b41063d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009ca0 tx=0x7fb7ac02a940 comp rx=0 tx=0).stop 2024-09-16T01:08:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 -- 172.21.15.78:0/2630681443 shutdown_connections 2024-09-16T01:08:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 --2- 172.21.15.78:0/2630681443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b41063d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 --2- 172.21.15.78:0/2630681443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4101450 0x7fb7b4105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 --2- 172.21.15.78:0/2630681443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4100b30 0x7fb7b4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 -- 172.21.15.78:0/2630681443 >> 172.21.15.78:0/2630681443 conn(0x7fb7b40fc7a0 msgr2=0x7fb7b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 -- 172.21.15.78:0/2630681443 shutdown_connections 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.205+0000 7fb7bba33640 1 -- 172.21.15.78:0/2630681443 wait complete. 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.206+0000 7fb7bba33640 1 Processor -- start 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.206+0000 7fb7bba33640 1 -- start start 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 0x7fb7b41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4101450 0x7fb7b41a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b4118570 con 0x7fb7b4105f40 2024-09-16T01:08:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b41183f0 con 0x7fb7b4100b30 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b41186f0 con 0x7fb7b4101450 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42742/0 (socket says 172.21.15.78:42742) 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 -- 172.21.15.78:0/81882053 learned_addr learned my addr 172.21.15.78:0/81882053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b8fce640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4101450 0x7fb7b41a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b97cf640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 0x7fb7b41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 -- 172.21.15.78:0/81882053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4101450 msgr2=0x7fb7b41a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4101450 0x7fb7b41a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 -- 172.21.15.78:0/81882053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 msgr2=0x7fb7b41a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 0x7fb7b41a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 -- 172.21.15.78:0/81882053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b419ef60 con 0x7fb7b4105f40 2024-09-16T01:08:33.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b97cf640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 0x7fb7b41a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7b9fd0640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009dc0 tx=0x7fb7ac02fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac034040 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b419f1f0 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7ac003e50 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac039430 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.207+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b41ab4a0 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.208+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b406b070 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.209+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb7ac037020 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.210+0000 7fb7aa7fc640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 0x7fb7a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.210+0000 7fb7b97cf640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 0x7fb7a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.210+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8277+0+0 (secure 0 0 0) 0x7fb7ac0f0c50 con 0x7fb7b4105f40 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.211+0000 7fb7b97cf640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 0x7fb7a00a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0005fd0 tx=0x7fb7b0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:33.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.214+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb7ac0f6020 con 0x7fb7b4105f40 2024-09-16T01:08:33.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:33.387+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb7b41ab670 con 0x7fb7b4105f40 2024-09-16T01:08:33.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:33 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T01:08:33.886 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:08:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: 2024-09-16T01:08:33.578+0000 7f4402b5b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:08:34.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:33 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T01:08:34.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:33 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-16T01:08:34.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.749+0000 7fb7aa7fc640 1 -- 172.21.15.78:0/81882053 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v86) ==== 228+0+0 (secure 0 0 0) 0x7fb7ac0bd940 con 0x7fb7b4105f40 2024-09-16T01:08:34.751 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 msgr2=0x7fb7a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 0x7fb7a00a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0005fd0 tx=0x7fb7b0005ea0 comp rx=0 tx=0).stop 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 msgr2=0x7fb7b419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009dc0 tx=0x7fb7ac02fb60 comp rx=0 tx=0).stop 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 shutdown_connections 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb7a00a3240 0x7fb7a00a5700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7b4105f40 0x7fb7b419e7c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:34.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7b4101450 0x7fb7b41a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:34.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 --2- 172.21.15.78:0/81882053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7b4100b30 0x7fb7b41a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:34.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 >> 172.21.15.78:0/81882053 conn(0x7fb7b40fc7a0 msgr2=0x7fb7b41038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:34.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 shutdown_connections 2024-09-16T01:08:34.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:34.753+0000 7fb7bba33640 1 -- 172.21.15.78:0/81882053 wait complete. 2024-09-16T01:08:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:34 smithi078 ceph-mon[30937]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T01:08:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:34 smithi078 ceph-mon[30937]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-16T01:08:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:34 smithi193 ceph-mon[35701]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T01:08:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:34 smithi193 ceph-mon[35701]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-16T01:08:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:34 smithi148 ceph-mon[35030]: osdmap e85: 12 total, 12 up, 12 in 2024-09-16T01:08:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:34 smithi148 ceph-mon[35030]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2024-09-16T01:08:35.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-16T01:08:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:35.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:35 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T01:08:35.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:35 smithi078 ceph-mon[30937]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T01:08:36.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:35 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T01:08:36.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:35 smithi193 ceph-mon[35701]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T01:08:36.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:35 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/81882053' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-16T01:08:36.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:35 smithi148 ceph-mon[35030]: osdmap e86: 12 total, 12 up, 12 in 2024-09-16T01:08:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 -- 172.21.15.78:0/4054587257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90101420 msgr2=0x7fbd9010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 --2- 172.21.15.78:0/4054587257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90101420 0x7fbd9010d950 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009cd0 tx=0x7fbd8802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 -- 172.21.15.78:0/4054587257 shutdown_connections 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 --2- 172.21.15.78:0/4054587257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90101420 0x7fbd9010d950 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 --2- 172.21.15.78:0/4054587257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90100a80 0x7fbd90100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 --2- 172.21.15.78:0/4054587257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90106a80 0x7fbd90106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 -- 172.21.15.78:0/4054587257 >> 172.21.15.78:0/4054587257 conn(0x7fbd900fc7a0 msgr2=0x7fbd900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 -- 172.21.15.78:0/4054587257 shutdown_connections 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.242+0000 7fbd96712640 1 -- 172.21.15.78:0/4054587257 wait complete. 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 Processor -- start 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 -- start start 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 0x7fbd900fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90101420 0x7fbd900fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90110ea0 con 0x7fbd90106a80 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90110d20 con 0x7fbd90100a80 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd96712640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90111020 con 0x7fbd90101420 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57354/0 (socket says 172.21.15.78:57354) 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 -- 172.21.15.78:0/867849130 learned_addr learned my addr 172.21.15.78:0/867849130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd8f7fe640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90101420 0x7fbd900fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd8ffff640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 0x7fbd900fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 -- 172.21.15.78:0/867849130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90101420 msgr2=0x7fbd900fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90101420 0x7fbd900fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 -- 172.21.15.78:0/867849130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 msgr2=0x7fbd900fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 0x7fbd900fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd94caf640 1 -- 172.21.15.78:0/867849130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901a68f0 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.243+0000 7fbd8ffff640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 0x7fbd900fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd94caf640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd88006010 tx=0x7fbd88003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88004450 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd88003e10 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88046480 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901a6b80 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.244+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd901a70c0 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.245+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd90100ee0 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.247+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbd880045f0 con 0x7fbd90106a80 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.247+0000 7fbd8d7fa640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 0x7fbd780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.248+0000 7fbd8ffff640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 0x7fbd780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.248+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fbd8802e030 con 0x7fbd90106a80 2024-09-16T01:08:36.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.248+0000 7fbd8ffff640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 0x7fbd780a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd800058c0 tx=0x7fbd80005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:36.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.251+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbd8804b060 con 0x7fbd90106a80 2024-09-16T01:08:36.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.420+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fbd900fe310 con 0x7fbd90106a80 2024-09-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.764+0000 7fbd8d7fa640 1 -- 172.21.15.78:0/867849130 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fbd900fe310 con 0x7fbd90106a80 2024-09-16T01:08:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2024-09-16T01:08:36.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 msgr2=0x7fbd780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:36.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 0x7fbd780a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd800058c0 tx=0x7fbd80005d30 comp rx=0 tx=0).stop 2024-09-16T01:08:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 msgr2=0x7fbd900fd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd88006010 tx=0x7fbd88003c20 comp rx=0 tx=0).stop 2024-09-16T01:08:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 shutdown_connections 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fbd780a3240 0x7fbd780a5700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd90106a80 0x7fbd900fd860 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd90101420 0x7fbd900fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 --2- 172.21.15.78:0/867849130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd90100a80 0x7fbd900fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 >> 172.21.15.78:0/867849130 conn(0x7fbd900fc7a0 msgr2=0x7fbd9006a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 shutdown_connections 2024-09-16T01:08:36.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:36.767+0000 7fbd96712640 1 -- 172.21.15.78:0/867849130 wait complete. 2024-09-16T01:08:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:36 smithi078 ceph-mon[30937]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T01:08:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:36 smithi078 ceph-mon[30937]: pgmap v286: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-16T01:08:37.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:36 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T01:08:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:36 smithi193 ceph-mon[35701]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T01:08:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:36 smithi193 ceph-mon[35701]: pgmap v286: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-16T01:08:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:36 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T01:08:37.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:36 smithi148 ceph-mon[35030]: osdmap e87: 12 total, 12 up, 12 in 2024-09-16T01:08:37.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:36 smithi148 ceph-mon[35030]: pgmap v286: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-16T01:08:37.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:36 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-16T01:08:37.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-16T01:08:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:38.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:37 smithi078 ceph-mon[30937]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:38.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:37 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T01:08:38.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:37 smithi078 ceph-mon[30937]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T01:08:38.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:37 smithi193 ceph-mon[35701]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:38.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:37 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T01:08:38.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:37 smithi193 ceph-mon[35701]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T01:08:38.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:37 smithi148 ceph-mon[35030]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T01:08:38.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:37 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/867849130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-16T01:08:38.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:37 smithi148 ceph-mon[35030]: osdmap e88: 12 total, 12 up, 12 in 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 -- 172.21.15.78:0/3600235003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc101420 msgr2=0x7ff9fc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3600235003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc101420 0x7ff9fc10d950 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009cd0 tx=0x7ff9f402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 -- 172.21.15.78:0/3600235003 shutdown_connections 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3600235003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc101420 0x7ff9fc10d950 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3600235003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc100a80 0x7ff9fc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3600235003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc106a80 0x7ff9fc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 -- 172.21.15.78:0/3600235003 >> 172.21.15.78:0/3600235003 conn(0x7ff9fc0fc7a0 msgr2=0x7ff9fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 -- 172.21.15.78:0/3600235003 shutdown_connections 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.226+0000 7ffa00a40640 1 -- 172.21.15.78:0/3600235003 wait complete. 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 Processor -- start 2024-09-16T01:08:38.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 -- start start 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc101420 0x7ff9fc1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 0x7ff9fc19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc110e80 con 0x7ff9fc100a80 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc110d00 con 0x7ff9fc106a80 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ffa00a40640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc111000 con 0x7ff9fc101420 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ff9fa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ff9fa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57382/0 (socket says 172.21.15.78:57382) 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ff9fa59c640 1 -- 172.21.15.78:0/3953897309 learned_addr learned my addr 172.21.15.78:0/3953897309 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.227+0000 7ff9fad9d640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 0x7ff9fc19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9f9d9b640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc101420 0x7ff9fc1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:38.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 -- 172.21.15.78:0/3953897309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc101420 msgr2=0x7ff9fc1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc101420 0x7ff9fc1988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 -- 172.21.15.78:0/3953897309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 msgr2=0x7ff9fc19cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 0x7ff9fc19cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 -- 172.21.15.78:0/3953897309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc19d340 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fad9d640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 0x7ff9fc19cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9fa59c640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d770 tx=0x7ff9ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec017070 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9ec005500 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc19d630 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec004e00 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.228+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc1a4f10 con 0x7ff9fc100a80 2024-09-16T01:08:38.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.229+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc100ee0 con 0x7ff9fc100a80 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.230+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff9ec004fa0 con 0x7ff9fc100a80 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.231+0000 7ff9eb7fe640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 0x7ff9e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.231+0000 7ff9f9d9b640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 0x7ff9e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.231+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7ff9ec0cf5b0 con 0x7ff9fc100a80 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.231+0000 7ff9f9d9b640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 0x7ff9e40a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc06b870 tx=0x7ff9f0005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:38.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.235+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff9ec09c0b0 con 0x7ff9fc100a80 2024-09-16T01:08:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.436+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7ff9fc1a51a0 con 0x7ff9fc100a80 2024-09-16T01:08:38.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.774+0000 7ff9eb7fe640 1 -- 172.21.15.78:0/3953897309 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7ff9ec09b9b0 con 0x7ff9fc100a80 2024-09-16T01:08:38.775 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 msgr2=0x7ff9e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 0x7ff9e40a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc06b870 tx=0x7ff9f0005cc0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 msgr2=0x7ff9fc198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d770 tx=0x7ff9ec00dc30 comp rx=0 tx=0).stop 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 shutdown_connections 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff9e40a3240 0x7ff9e40a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9fc106a80 0x7ff9fc19cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9fc101420 0x7ff9fc1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 --2- 172.21.15.78:0/3953897309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9fc100a80 0x7ff9fc198390 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:38.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 >> 172.21.15.78:0/3953897309 conn(0x7ff9fc0fc7a0 msgr2=0x7ff9fc104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:38.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 shutdown_connections 2024-09-16T01:08:38.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:38.776+0000 7ffa00a40640 1 -- 172.21.15.78:0/3953897309 wait complete. 2024-09-16T01:08:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:38 smithi078 ceph-mon[30937]: pgmap v288: 129 pgs: 4 creating+activating, 4 unknown, 121 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T01:08:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:38 smithi193 ceph-mon[35701]: pgmap v288: 129 pgs: 4 creating+activating, 4 unknown, 121 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T01:08:39.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:38 smithi148 ceph-mon[35030]: pgmap v288: 129 pgs: 4 creating+activating, 4 unknown, 121 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:39.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-16T01:08:39.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:39.728 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:40.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:39 smithi078 ceph-mon[30937]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T01:08:40.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:39 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T01:08:40.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:39 smithi078 ceph-mon[30937]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:40.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:39 smithi193 ceph-mon[35701]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T01:08:40.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:39 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T01:08:40.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:39 smithi193 ceph-mon[35701]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:40.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:39 smithi148 ceph-mon[35030]: osdmap e89: 12 total, 12 up, 12 in 2024-09-16T01:08:40.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:39 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3953897309' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-16T01:08:40.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:39 smithi148 ceph-mon[35030]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:08:40.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- 172.21.15.78:0/3609595538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4101420 msgr2=0x7f04d410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/3609595538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4101420 0x7f04d410d950 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009cd0 tx=0x7f04cc02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- 172.21.15.78:0/3609595538 shutdown_connections 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/3609595538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4101420 0x7f04d410d950 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/3609595538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4100a80 0x7f04d4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/3609595538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4106a80 0x7f04d4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- 172.21.15.78:0/3609595538 >> 172.21.15.78:0/3609595538 conn(0x7f04d40fc7a0 msgr2=0x7f04d40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- 172.21.15.78:0/3609595538 shutdown_connections 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- 172.21.15.78:0/3609595538 wait complete. 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 Processor -- start 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.281+0000 7f04d9fc3640 1 -- start start 2024-09-16T01:08:40.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4100a80 0x7f04d4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4101420 0x7f04d4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4111fa0 con 0x7f04d4106a80 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4111e20 con 0x7f04d4101420 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4112120 con 0x7f04d4100a80 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57404/0 (socket says 172.21.15.78:57404) 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 -- 172.21.15.78:0/2914674285 learned_addr learned my addr 172.21.15.78:0/2914674285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d37fe640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4100a80 0x7f04d4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04cbfff640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4101420 0x7f04d4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 -- 172.21.15.78:0/2914674285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4100a80 msgr2=0x7f04d4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4100a80 0x7f04d4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 -- 172.21.15.78:0/2914674285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4101420 msgr2=0x7f04d4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4101420 0x7f04d4198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 -- 172.21.15.78:0/2914674285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d419d3b0 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d3fff640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04cc006010 tx=0x7f04cc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc004450 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d419d640 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04cc003e10 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0466e0 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.282+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d41a4f80 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.283+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d4100ee0 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.284+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f04cc0305a0 con 0x7f04d4106a80 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.285+0000 7f04d17fa640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 0x7f04c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.285+0000 7f04d37fe640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 0x7f04c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.285+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f04cc02e030 con 0x7f04d4106a80 2024-09-16T01:08:40.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.285+0000 7f04d37fe640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 0x7f04c00a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009890 tx=0x7f04c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:40.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.289+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f04cc04b060 con 0x7f04d4106a80 2024-09-16T01:08:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.488+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04d41a5150 con 0x7f04d4106a80 2024-09-16T01:08:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.489+0000 7f04d17fa640 1 -- 172.21.15.78:0/2914674285 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11845 (secure 0 0 0) 0x7f04cc039380 con 0x7f04d4106a80 2024-09-16T01:08:40.491 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:40.491 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"btime":"2024-09-16T01:08:38:766267+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:08:38.437867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24485,"qdb_cluster":[24485]},"id":1}]} 2024-09-16T01:08:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 16 2024-09-16T01:08:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 msgr2=0x7f04c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 0x7f04c00a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009890 tx=0x7f04c4008040 comp rx=0 tx=0).stop 2024-09-16T01:08:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 msgr2=0x7f04d419ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04cc006010 tx=0x7f04cc003c20 comp rx=0 tx=0).stop 2024-09-16T01:08:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 shutdown_connections 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04c00a3240 0x7f04c00a5700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04d4106a80 0x7f04d419ccd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04d4101420 0x7f04d4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 --2- 172.21.15.78:0/2914674285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04d4100a80 0x7f04d4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 >> 172.21.15.78:0/2914674285 conn(0x7f04d40fc7a0 msgr2=0x7f04d40fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 shutdown_connections 2024-09-16T01:08:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:40.491+0000 7f04d9fc3640 1 -- 172.21.15.78:0/2914674285 wait complete. 2024-09-16T01:08:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:40 smithi193 ceph-mon[35701]: pgmap v290: 129 pgs: 4 creating+activating, 125 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:40 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2914674285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:41.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:40 smithi078 ceph-mon[30937]: pgmap v290: 129 pgs: 4 creating+activating, 125 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:41.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:40 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2914674285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:41.102 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:08:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:40 smithi148 ceph-mon[35030]: pgmap v290: 129 pgs: 4 creating+activating, 125 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:40 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2914674285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 -- 172.21.15.78:0/147164076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 msgr2=0x7fdae01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 --2- 172.21.15.78:0/147164076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae01067e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009900 tx=0x7fdacc02a410 comp rx=0 tx=0).stop 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 -- 172.21.15.78:0/147164076 shutdown_connections 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 --2- 172.21.15.78:0/147164076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0106f10 0x7fdae010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 --2- 172.21.15.78:0/147164076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae01067e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 --2- 172.21.15.78:0/147164076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0101de0 0x7fdae01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 -- 172.21.15.78:0/147164076 >> 172.21.15.78:0/147164076 conn(0x7fdae00fd8c0 msgr2=0x7fdae00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 -- 172.21.15.78:0/147164076 shutdown_connections 2024-09-16T01:08:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.840+0000 7fdae6c14640 1 -- 172.21.15.78:0/147164076 wait complete. 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 Processor -- start 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 -- start start 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0101de0 0x7fdae0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0106f10 0x7fdae019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0111f70 con 0x7fdae0102790 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0111df0 con 0x7fdae0106f10 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae6c14640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdae01120f0 con 0x7fdae0101de0 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57428/0 (socket says 172.21.15.78:57428) 2024-09-16T01:08:41.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 -- 172.21.15.78:0/1085469534 learned_addr learned my addr 172.21.15.78:0/1085469534 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae49b0640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0101de0 0x7fdae0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdae51b1640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0106f10 0x7fdae019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 -- 172.21.15.78:0/1085469534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0101de0 msgr2=0x7fdae0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0101de0 0x7fdae0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 -- 172.21.15.78:0/1085469534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0106f10 msgr2=0x7fdae019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0106f10 0x7fdae019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.841+0000 7fdadffff640 1 -- 172.21.15.78:0/1085469534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae019d3e0 con 0x7fdae0102790 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdadffff640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdacc005ab0 tx=0x7fdacc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc03f030 con 0x7fdae0102790 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdacc003ce0 con 0x7fdae0102790 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae019d670 con 0x7fdae0102790 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc035070 con 0x7fdae0102790 2024-09-16T01:08:41.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.842+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae01a4fb0 con 0x7fdae0102790 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.843+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae01021c0 con 0x7fdae0102790 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.844+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdacc042030 con 0x7fdae0102790 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.845+0000 7fdaddffb640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 0x7fdad00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.845+0000 7fdae49b0640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 0x7fdad00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.845+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdacc0f0690 con 0x7fdae0102790 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.845+0000 7fdae49b0640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 0x7fdad00a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdad40058c0 tx=0x7fdad400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:41.848+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdacc0bd170 con 0x7fdae0102790 2024-09-16T01:08:42.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.018+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdae010a0f0 con 0x7fdae0102790 2024-09-16T01:08:42.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:41 smithi078 ceph-mon[30937]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T01:08:42.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:41 smithi078 ceph-mon[30937]: Cluster is now healthy 2024-09-16T01:08:42.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.019+0000 7fdaddffb640 1 -- 172.21.15.78:0/1085469534 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7fdacc0bca70 con 0x7fdae0102790 2024-09-16T01:08:42.022 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:42.022 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.021+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 msgr2=0x7fdad00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.021+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 0x7fdad00a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdad40058c0 tx=0x7fdad400a2d0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.021+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 msgr2=0x7fdae0198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.021+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdacc005ab0 tx=0x7fdacc0042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 shutdown_connections 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdad00a3240 0x7fdad00a5700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdae0106f10 0x7fdae019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdae0102790 0x7fdae0198970 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 --2- 172.21.15.78:0/1085469534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae0101de0 0x7fdae0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:42.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 >> 172.21.15.78:0/1085469534 conn(0x7fdae00fd8c0 msgr2=0x7fdae00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:42.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 shutdown_connections 2024-09-16T01:08:42.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:42.022+0000 7fdae6c14640 1 -- 172.21.15.78:0/1085469534 wait complete. 2024-09-16T01:08:42.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:41 smithi193 ceph-mon[35701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T01:08:42.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:41 smithi193 ceph-mon[35701]: Cluster is now healthy 2024-09-16T01:08:42.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:41 smithi148 ceph-mon[35030]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-16T01:08:42.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:41 smithi148 ceph-mon[35030]: Cluster is now healthy 2024-09-16T01:08:42.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:08:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:42 smithi193 ceph-mon[35701]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1085469534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:43.092 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:42 smithi078 ceph-mon[30937]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:43.093 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1085469534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:43.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:42 smithi148 ceph-mon[35030]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:43.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1085469534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2390736326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 msgr2=0x7fd6e8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2390736326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e8100ee0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009930 tx=0x7fd6d002a420 comp rx=0 tx=0).stop 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2390736326 shutdown_connections 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2390736326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8101420 0x7fd6e810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2390736326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e8100ee0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2390736326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8106a80 0x7fd6e8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2390736326 >> 172.21.15.78:0/2390736326 conn(0x7fd6e80fc7a0 msgr2=0x7fd6e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2390736326 shutdown_connections 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2390736326 wait complete. 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.475+0000 7fd6ed9a5640 1 Processor -- start 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 -- start start 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 0x7fd6e8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8106a80 0x7fd6e819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8110f10 con 0x7fd6e8100a80 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8110d90 con 0x7fd6e8101420 2024-09-16T01:08:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6ed9a5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8111090 con 0x7fd6e8106a80 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57448/0 (socket says 172.21.15.78:57448) 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 -- 172.21.15.78:0/2962216084 learned_addr learned my addr 172.21.15.78:0/2962216084 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e67fc640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 0x7fd6e8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 -- 172.21.15.78:0/2962216084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8106a80 msgr2=0x7fd6e819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8106a80 0x7fd6e819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 -- 172.21.15.78:0/2962216084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 msgr2=0x7fd6e8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 0x7fd6e8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e6ffd640 1 -- 172.21.15.78:0/2962216084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e819d480 con 0x7fd6e8100a80 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.476+0000 7fd6e67fc640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 0x7fd6e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6e6ffd640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b4a0 tx=0x7fd6d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8014070 con 0x7fd6e8100a80 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e819d770 con 0x7fd6e8100a80 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8010af0 con 0x7fd6e8100a80 2024-09-16T01:08:43.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8017d10 con 0x7fd6e8100a80 2024-09-16T01:08:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.477+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e81a5050 con 0x7fd6e8100a80 2024-09-16T01:08:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.478+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e8100ee0 con 0x7fd6e8100a80 2024-09-16T01:08:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.479+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd6d8017070 con 0x7fd6e8100a80 2024-09-16T01:08:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.479+0000 7fd6ec9a3640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 0x7fd6d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.480+0000 7fd6e67fc640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 0x7fd6d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.480+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6d80cefd0 con 0x7fd6e8100a80 2024-09-16T01:08:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.480+0000 7fd6e67fc640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 0x7fd6d40a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6d002a940 tx=0x7fd6d0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.484+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6d809bab0 con 0x7fd6e8100a80 2024-09-16T01:08:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.651+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6e8199830 con 0x7fd6e8100a80 2024-09-16T01:08:43.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.653+0000 7fd6ec9a3640 1 -- 172.21.15.78:0/2962216084 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7fd6d809b3b0 con 0x7fd6e8100a80 2024-09-16T01:08:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:43.656 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 msgr2=0x7fd6d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 0x7fd6d40a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6d002a940 tx=0x7fd6d0005fd0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 msgr2=0x7fd6e81984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b4a0 tx=0x7fd6d800b960 comp rx=0 tx=0).stop 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 shutdown_connections 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6d40a3240 0x7fd6d40a5700 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8106a80 0x7fd6e819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6e8101420 0x7fd6e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 --2- 172.21.15.78:0/2962216084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6e8100a80 0x7fd6e81984d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 >> 172.21.15.78:0/2962216084 conn(0x7fd6e80fc7a0 msgr2=0x7fd6e81047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 shutdown_connections 2024-09-16T01:08:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:43.655+0000 7fd6ed9a5640 1 -- 172.21.15.78:0/2962216084 wait complete. 2024-09-16T01:08:44.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2962216084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:44.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2962216084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:44.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2962216084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:44.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:08:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:44 smithi148 ceph-mon[35030]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:45.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:44 smithi078 ceph-mon[30937]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 -- 172.21.15.78:0/148045746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100310 msgr2=0x7fac801006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 --2- 172.21.15.78:0/148045746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100310 0x7fac801006f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fac70009900 tx=0x7fac7002a3f0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 -- 172.21.15.78:0/148045746 shutdown_connections 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 --2- 172.21.15.78:0/148045746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 0x7fac8010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 --2- 172.21.15.78:0/148045746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100cc0 0x7fac80106800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 --2- 172.21.15.78:0/148045746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100310 0x7fac801006f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 -- 172.21.15.78:0/148045746 >> 172.21.15.78:0/148045746 conn(0x7fac80077210 msgr2=0x7fac800fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:45.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 -- 172.21.15.78:0/148045746 shutdown_connections 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 -- 172.21.15.78:0/148045746 wait complete. 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.226+0000 7fac859bb640 1 Processor -- start 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 -- start start 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 0x7fac80101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 0x7fac80101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac80111fb0 con 0x7fac80100cc0 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fac80111e30 con 0x7fac80100310 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac859bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac80112130 con 0x7fac80106f30 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34300/0 (socket says 172.21.15.78:34300) 2024-09-16T01:08:45.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 -- 172.21.15.78:0/2350900618 learned_addr learned my addr 172.21.15.78:0/2350900618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 0x7fac80101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7effd640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 0x7fac80101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 -- 172.21.15.78:0/2350900618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 msgr2=0x7fac80101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 0x7fac80101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 -- 172.21.15.78:0/2350900618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 msgr2=0x7fac80101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 0x7fac80101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7e7fc640 1 -- 172.21.15.78:0/2350900618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac801a6970 con 0x7fac80100cc0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.227+0000 7fac7effd640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 0x7fac80101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac7e7fc640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac6800b4a0 tx=0x7fac6800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68014070 con 0x7fac80100cc0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac68010af0 con 0x7fac80100cc0 2024-09-16T01:08:45.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac801a6c60 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68017d10 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.228+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac801a71a0 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.229+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac801006f0 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.230+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fac68017070 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.231+0000 7fac849b9640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 0x7fac6c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.231+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac680cef70 con 0x7fac80100cc0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.231+0000 7fac7effd640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 0x7fac6c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:45.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.231+0000 7fac7effd640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 0x7fac6c0a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac70000c00 tx=0x7fac70032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:45.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.235+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac6809ba50 con 0x7fac80100cc0 2024-09-16T01:08:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:44 smithi193 ceph-mon[35701]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:45.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.434+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac8010a110 con 0x7fac80100cc0 2024-09-16T01:08:45.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.435+0000 7fac849b9640 1 -- 172.21.15.78:0/2350900618 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11845 (secure 0 0 0) 0x7fac6809b350 con 0x7fac80100cc0 2024-09-16T01:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:45.438 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"btime":"2024-09-16T01:08:38:766267+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24436,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:08:38.437867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24485,"qdb_cluster":[24485]},"id":1}]} 2024-09-16T01:08:45.438 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 16 2024-09-16T01:08:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 msgr2=0x7fac6c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:45.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 0x7fac6c0a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac70000c00 tx=0x7fac70032d50 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 msgr2=0x7fac801017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fac6800b4a0 tx=0x7fac6800b960 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 shutdown_connections 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fac6c0a3240 0x7fac6c0a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac80106f30 0x7fac80101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac80100cc0 0x7fac801017d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.437+0000 7fac859bb640 1 --2- 172.21.15.78:0/2350900618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac80100310 0x7fac80101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.438+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 >> 172.21.15.78:0/2350900618 conn(0x7fac80077210 msgr2=0x7fac801057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.438+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 shutdown_connections 2024-09-16T01:08:45.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:45.438+0000 7fac859bb640 1 -- 172.21.15.78:0/2350900618 wait complete. 2024-09-16T01:08:46.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:08:46.412 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:46.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:46 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2350900618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:46.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:46 smithi078 ceph-mon[30937]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:46.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:46 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2350900618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:46.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:46 smithi148 ceph-mon[35030]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:46.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:46 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2350900618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:08:46.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:46 smithi193 ceph-mon[35701]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- 172.21.15.78:0/3532112499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c106f10 msgr2=0x7fb13c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 --2- 172.21.15.78:0/3532112499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c106f10 0x7fb13c10ea90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb134009ca0 tx=0x7fb13402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- 172.21.15.78:0/3532112499 shutdown_connections 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 --2- 172.21.15.78:0/3532112499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c106f10 0x7fb13c10ea90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 --2- 172.21.15.78:0/3532112499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 0x7fb13c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 --2- 172.21.15.78:0/3532112499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c101de0 0x7fb13c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- 172.21.15.78:0/3532112499 >> 172.21.15.78:0/3532112499 conn(0x7fb13c0fd8c0 msgr2=0x7fb13c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- 172.21.15.78:0/3532112499 shutdown_connections 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- 172.21.15.78:0/3532112499 wait complete. 2024-09-16T01:08:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 Processor -- start 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.877+0000 7fb142e08640 1 -- start start 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 0x7fb13c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 0x7fb13c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c111f70 con 0x7fb13c101de0 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c111df0 con 0x7fb13c106f10 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1120f0 con 0x7fb13c102790 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34322/0 (socket says 172.21.15.78:34322) 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 -- 172.21.15.78:0/2855540711 learned_addr learned my addr 172.21.15.78:0/2855540711 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb13bfff640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 0x7fb13c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb1413a5640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 0x7fb13c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 -- 172.21.15.78:0/2855540711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 msgr2=0x7fb13c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 0x7fb13c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 -- 172.21.15.78:0/2855540711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 msgr2=0x7fb13c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 0x7fb13c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 -- 172.21.15.78:0/2855540711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c19d3c0 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb1413a5640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 0x7fb13c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb140ba4640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c890 tx=0x7fb13000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130004220 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c19d6b0 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1300043c0 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.878+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130004a00 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.879+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a4f90 con 0x7fb13c101de0 2024-09-16T01:08:46.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.880+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c1021c0 con 0x7fb13c101de0 2024-09-16T01:08:46.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.881+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb1300079b0 con 0x7fb13c101de0 2024-09-16T01:08:46.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.881+0000 7fb139ffb640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 0x7fb12c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:46.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.881+0000 7fb13bfff640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 0x7fb12c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.881+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1300cf850 con 0x7fb13c101de0 2024-09-16T01:08:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.882+0000 7fb13bfff640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 0x7fb12c0a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0fdce0 tx=0x7fb12800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:46.886+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb130014030 con 0x7fb13c101de0 2024-09-16T01:08:47.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.055+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb13c10a1b0 con 0x7fb13c101de0 2024-09-16T01:08:47.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.057+0000 7fb139ffb640 1 -- 172.21.15.78:0/2855540711 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7fb13009c330 con 0x7fb13c101de0 2024-09-16T01:08:47.059 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:47.059 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:08:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 msgr2=0x7fb12c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 0x7fb12c0a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0fdce0 tx=0x7fb12800b390 comp rx=0 tx=0).stop 2024-09-16T01:08:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 msgr2=0x7fb13c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c890 tx=0x7fb13000cd50 comp rx=0 tx=0).stop 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 shutdown_connections 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb12c0a3240 0x7fb12c0a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb13c106f10 0x7fb13c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c102790 0x7fb13c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 --2- 172.21.15.78:0/2855540711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb13c101de0 0x7fb13c198410 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 >> 172.21.15.78:0/2855540711 conn(0x7fb13c0fd8c0 msgr2=0x7fb13c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 shutdown_connections 2024-09-16T01:08:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:47.059+0000 7fb142e08640 1 -- 172.21.15.78:0/2855540711 wait complete. 2024-09-16T01:08:47.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:47 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2855540711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:47.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:47 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2855540711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:47.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:47 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2855540711' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:47.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:08:29.385380+0000', 'modified': '2024-09-16T01:08:38.437867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24485}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6839/3106047695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 3106047695}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 3106047695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24485, 'qdb_cluster': [24485]} 2024-09-16T01:08:47.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:08:47.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T01:08:48.169 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:48 smithi078 ceph-mon[30937]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:48.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:48 smithi148 ceph-mon[35030]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 -- 172.21.15.78:0/1285840787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 msgr2=0x7f35901067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 --2- 172.21.15.78:0/1285840787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901067e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3580009900 tx=0x7f358002a410 comp rx=0 tx=0).stop 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 -- 172.21.15.78:0/1285840787 shutdown_connections 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 --2- 172.21.15.78:0/1285840787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 0x7f359010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 --2- 172.21.15.78:0/1285840787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901067e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 --2- 172.21.15.78:0/1285840787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f35901021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 -- 172.21.15.78:0/1285840787 >> 172.21.15.78:0/1285840787 conn(0x7f35900fd8c0 msgr2=0x7f35900ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 -- 172.21.15.78:0/1285840787 shutdown_connections 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.654+0000 7f3598293640 1 -- 172.21.15.78:0/1285840787 wait complete. 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 Processor -- start 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 -- start start 2024-09-16T01:08:48.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f3590198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 0x7f359019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3590111f70 con 0x7f3590102790 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3590111df0 con 0x7f3590101de0 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3598293640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35901120f0 con 0x7f3590106f10 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34334/0 (socket says 172.21.15.78:34334) 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 -- 172.21.15.78:0/3000130805 learned_addr learned my addr 172.21.15.78:0/3000130805 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:48.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f3596830640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 0x7f359019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359602f640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f3590198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 -- 172.21.15.78:0/3000130805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 msgr2=0x7f359019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 0x7f359019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 -- 172.21.15.78:0/3000130805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 msgr2=0x7f3590198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f3590198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359582e640 1 -- 172.21.15.78:0/3000130805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359019d410 con 0x7f3590102790 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.655+0000 7f359602f640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f3590198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f359582e640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f35800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f35877fe640 1 -- 172.21.15.78:0/3000130805 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358003f030 con 0x7f3590102790 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f35877fe640 1 -- 172.21.15.78:0/3000130805 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3580003ce0 con 0x7f3590102790 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f35877fe640 1 -- 172.21.15.78:0/3000130805 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580035070 con 0x7f3590102790 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359019d6a0 con 0x7f3590102790 2024-09-16T01:08:48.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.656+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f359019db30 con 0x7f3590102790 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.658+0000 7f35877fe640 1 -- 172.21.15.78:0/3000130805 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3580042030 con 0x7f3590102790 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.658+0000 7f359a66c740 1 Processor -- start 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.658+0000 7f359a66c740 1 -- start start 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 0x5555dbf87db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf94fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5555dbeee090 con 0x5555dbf882f0 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5555dbe95a70 con 0x5555dbf879d0 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359a66c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5555dbeea050 con 0x5555dbf92bb0 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34342/0 (socket says 172.21.15.78:34342) 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 -- 172.21.15.78:0/3881333351 learned_addr learned my addr 172.21.15.78:0/3881333351 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:48.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f35877fe640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 0x7f357c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359582e640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf94fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f3596830640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 0x7f357c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f3596830640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 0x5555dbf87db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 -- 172.21.15.78:0/3881333351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 msgr2=0x5555dbf94fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf94fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 -- 172.21.15.78:0/3881333351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 msgr2=0x5555dbf87db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 0x5555dbf87db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 -- 172.21.15.78:0/3881333351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555dbea2130 con 0x5555dbf882f0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f3596830640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 0x5555dbf87db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f35877fe640 1 -- 172.21.15.78:0/3000130805 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35800f00f0 con 0x7f3590102790 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.659+0000 7f359602f640 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f358c00bb70 tx=0x7f358c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13d97dc20862c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f3584dea640 1 -- 172.21.15.78:0/3881333351 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c017620 con 0x5555dbf882f0 2024-09-16T01:08:48.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f3584dea640 1 -- 172.21.15.78:0/3881333351 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358c017ca0 con 0x5555dbf882f0 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f3584dea640 1 -- 172.21.15.78:0/3881333351 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c00fd10 con 0x5555dbf882f0 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f3596830640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 0x7f357c0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35880098b0 tx=0x7f3588009840 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 -- 172.21.15.78:0/3881333351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 msgr2=0x5555dbf92670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f358c00bb70 tx=0x7f358c00d710 comp rx=0 tx=0).stop 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 -- 172.21.15.78:0/3881333351 shutdown_connections 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf94fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf882f0 0x5555dbf92670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 --2- 172.21.15.78:0/3881333351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf879d0 0x5555dbf87db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 -- 172.21.15.78:0/3881333351 >> 172.21.15.78:0/3881333351 conn(0x5555dbf83f60 msgr2=0x5555dbf84340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 -- 172.21.15.78:0/3881333351 shutdown_connections 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.660+0000 7f359a66c740 1 -- 172.21.15.78:0/3881333351 wait complete. 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 -- start start 2024-09-16T01:08:48.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 Processor -- start 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbf8f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf8fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5555dbf95780 con 0x5555dbf879d0 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5555dbea8f20 con 0x5555dbf882f0 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5555dbf95900 con 0x5555dbf92bb0 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359a66c740 10 client.0 _ll_register_callbacks cb 0x5555dbe72030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34346/0 (socket says 172.21.15.78:34346) 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 -- 172.21.15.78:0/340626708 learned_addr learned my addr 172.21.15.78:0/340626708 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 -- 172.21.15.78:0/340626708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 msgr2=0x5555dbf8fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf8fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 -- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 msgr2=0x5555dbf8f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbf8f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.661+0000 7f359602f640 1 -- 172.21.15.78:0/340626708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555dbc5f2b0 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359602f640 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f358c019040 tx=0x7f358c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0182b27252861f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c026070 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f358c017830 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f358c017ca0 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f358c032050 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 10 client.14700 fetch_fsmap learned FSMap version 16 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5555dbfa0f80 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358c017ca0 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f358c022a40 con 0x5555dbf879d0 2024-09-16T01:08:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 10 client.14700 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 10 client.14700 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 20 client.14700 populate_metadata read hostname 'smithi078' 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dbfa3260 0x5555dbfa46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 4 client.14700 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.662+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555dbefb600 con 0x5555dbfa3260 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.663+0000 7f359582e640 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dbfa3260 0x5555dbfa46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.663+0000 7f3579ffb640 10 client.14700 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.663+0000 7f359582e640 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dbfa3260 0x5555dbfa46d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e10c168e3d25c7f server_cookie=4d3bb378e01a21cc in_seq=0 out_seq=0 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f3579ffb640 1 -- 172.21.15.78:0/340626708 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5555dbefb600 con 0x5555dbfa3260 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f3579ffb640 10 client.14700 handle_command_reply: tid=0 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 client.14700 shutdown 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 msgr2=0x5555dbf923b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f358c019040 tx=0x7f358c00b8b0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 shutdown_connections 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dbfa3260 0x5555dbfa46d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbf8fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbf8f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 --2- 172.21.15.78:0/340626708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbf923b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f3579ffb640 0 client.14700 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 >> 172.21.15.78:0/340626708 conn(0x5555dbf83f60 msgr2=0x5555dbf89c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 shutdown_connections 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 1 -- 172.21.15.78:0/340626708 wait complete. 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.664+0000 7f359a66c740 20 client.14700 trim_cache size 0 max 16384 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 Processor -- start 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 -- start start 2024-09-16T01:08:48.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbfa4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbfa2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa2a10 con 0x5555dbf879d0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa30e0 con 0x5555dbf882f0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359a66c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa37b0 con 0x5555dbf92bb0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34350/0 (socket says 172.21.15.78:34350) 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 -- 172.21.15.78:0/891327060 learned_addr learned my addr 172.21.15.78:0/891327060 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f3596830640 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbfa2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359582e640 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbfa4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 -- 172.21.15.78:0/891327060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 msgr2=0x5555dbfa2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbfa2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 -- 172.21.15.78:0/891327060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 msgr2=0x5555dbfa4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbfa4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.682+0000 7f359602f640 1 -- 172.21.15.78:0/891327060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555dbf83a00 con 0x5555dbf879d0 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359602f640 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f358c020430 tx=0x7f358c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477efee4a725eb7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f357bfff640 1 -- 172.21.15.78:0/891327060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c03b070 con 0x5555dbf879d0 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f357bfff640 1 -- 172.21.15.78:0/891327060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f358c005b20 con 0x5555dbf879d0 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f357bfff640 1 -- 172.21.15.78:0/891327060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c02f030 con 0x5555dbf879d0 2024-09-16T01:08:48.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 -- 172.21.15.78:0/891327060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 msgr2=0x5555dbfa3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f358c020430 tx=0x7f358c00dd40 comp rx=0 tx=0).stop 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 -- 172.21.15.78:0/891327060 shutdown_connections 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf92bb0 0x5555dbfa2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf882f0 0x5555dbfa4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 --2- 172.21.15.78:0/891327060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa3260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 -- 172.21.15.78:0/891327060 >> 172.21.15.78:0/891327060 conn(0x5555dbf83f60 msgr2=0x5555dbf8a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 -- 172.21.15.78:0/891327060 shutdown_connections 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.683+0000 7f359a66c740 1 -- 172.21.15.78:0/891327060 wait complete. 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 -- start start 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 Processor -- start 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf882f0 0x5555dbfa5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf92bb0 0x5555dbfa6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa4550 con 0x5555dbf879d0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa3e80 con 0x5555dbf92bb0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5555dbfa4c20 con 0x5555dbf882f0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359a66c740 10 client.0 _ll_register_callbacks cb 0x7f35800d1380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34362/0 (socket says 172.21.15.78:34362) 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 -- 172.21.15.78:0/1233568528 learned_addr learned my addr 172.21.15.78:0/1233568528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359582e640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf882f0 0x5555dbfa5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 -- 172.21.15.78:0/1233568528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf882f0 msgr2=0x5555dbfa5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf882f0 0x5555dbfa5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f3596830640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf92bb0 0x5555dbfa6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 -- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf92bb0 msgr2=0x5555dbfa6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf92bb0 0x5555dbfa6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.684+0000 7f359602f640 1 -- 172.21.15.78:0/1233568528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555dc00b960 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359602f640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f358c022de0 tx=0x7f358c00fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5608fcd578d466 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c03b070 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f358c020950 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5555dbef5b10 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f358c02f040 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 10 client.14706 fetch_fsmap learned FSMap version 16 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5555dc00cc40 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5555dbef5b10 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f358c017630 con 0x5555dbf879d0 2024-09-16T01:08:48.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 10 client.14706 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 10 client.14706 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 20 client.14706 populate_metadata read hostname 'smithi078' 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dc0114d0 0x5555dc0101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 4 client.14706 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.685+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "status"}) -- 0x5555dbe720e0 con 0x5555dc0114d0 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359582e640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dc0114d0 0x5555dc0101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f3579ffb640 10 client.14706 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359582e640 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dc0114d0 0x5555dc0101e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b52b2fc0f7ee57d0 server_cookie=954ec85f79512ecd in_seq=0 out_seq=0 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f3579ffb640 1 -- 172.21.15.78:0/1233568528 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x5555dbe720e0 con 0x5555dc0114d0 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f3579ffb640 10 client.14706 handle_command_reply: tid=0 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 client.14706 shutdown 2024-09-16T01:08:48.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 msgr2=0x5555dbfa57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f358c022de0 tx=0x7f358c00fda0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 shutdown_connections 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5555dc0114d0 0x5555dc0101e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5555dbf92bb0 0x5555dbfa6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555dbf882f0 0x5555dbfa5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f3579ffb640 0 client.14706 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 --2- 172.21.15.78:0/1233568528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5555dbf879d0 0x5555dbfa57e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 >> 172.21.15.78:0/1233568528 conn(0x5555dbf83f60 msgr2=0x5555dbf8a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 shutdown_connections 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.686+0000 7f359a66c740 1 -- 172.21.15.78:0/1233568528 wait complete. 2024-09-16T01:08:48.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f359a66c740 20 client.14706 trim_cache size 0 max 16384 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 18.95653158, 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 16, 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 89, 2024-09-16T01:08:48.706 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:08:48.707 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 30.915006282 2024-09-16T01:08:48.707 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:08:48.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 msgr2=0x7f357c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 0x7f357c0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f35880098b0 tx=0x7f3588009840 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 msgr2=0x7f35901989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3580006010 tx=0x7f35800042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 shutdown_connections 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f357c0a3240 0x7f357c0a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3590106f10 0x7f359019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3590102790 0x7f35901989a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 --2- 172.21.15.78:0/3000130805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3590101de0 0x7f3590198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 >> 172.21.15.78:0/3000130805 conn(0x7f35900fd8c0 msgr2=0x7f35900fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 shutdown_connections 2024-09-16T01:08:48.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:48.687+0000 7f3598293640 1 -- 172.21.15.78:0/3000130805 wait complete. 2024-09-16T01:08:48.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:48 smithi193 ceph-mon[35701]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:49.543 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:08:49.543 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:49.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-16T01:08:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 -- 172.21.15.78:0/1112443082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 msgr2=0x7fe96410b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 --2- 172.21.15.78:0/1112443082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe96410b9a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe958009900 tx=0x7fe95802a480 comp rx=0 tx=0).stop 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 -- 172.21.15.78:0/1112443082 shutdown_connections 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 --2- 172.21.15.78:0/1112443082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe96410bfc0 0x7fe96410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 --2- 172.21.15.78:0/1112443082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe96410b9a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 --2- 172.21.15.78:0/1112443082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe964102b90 0x7fe964102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 -- 172.21.15.78:0/1112443082 >> 172.21.15.78:0/1112443082 conn(0x7fe9640fc630 msgr2=0x7fe9640fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 -- 172.21.15.78:0/1112443082 shutdown_connections 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.270+0000 7fe96979d640 1 -- 172.21.15.78:0/1112443082 wait complete. 2024-09-16T01:08:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 Processor -- start 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 -- start start 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe964102b90 0x7fe9641007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 0x7fe964101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe964111e20 con 0x7fe9641034b0 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe964111ca0 con 0x7fe96410bfc0 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe96979d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe964111fa0 con 0x7fe964102b90 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe9637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe9637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34372/0 (socket says 172.21.15.78:34372) 2024-09-16T01:08:50.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe9637fe640 1 -- 172.21.15.78:0/3527543499 learned_addr learned my addr 172.21.15.78:0/3527543499 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:50.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.271+0000 7fe963fff640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe964102b90 0x7fe9641007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:50.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe968f9c640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 0x7fe964101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:50.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 -- 172.21.15.78:0/3527543499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe964102b90 msgr2=0x7fe9641007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:50.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe964102b90 0x7fe9641007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 -- 172.21.15.78:0/3527543499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 msgr2=0x7fe964101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 0x7fe964101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 -- 172.21.15.78:0/3527543499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9641a67e0 con 0x7fe9641034b0 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe968f9c640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 0x7fe964101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9637fe640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe958006010 tx=0x7fe9580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95803f030 con 0x7fe9641034b0 2024-09-16T01:08:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9641a6a70 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe958003ce0 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe958035070 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.272+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9641a6fa0 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.273+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe964102f70 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.274+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe958042030 con 0x7fe9641034b0 2024-09-16T01:08:50.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.275+0000 7fe9617fa640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 0x7fe94c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:50.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.275+0000 7fe963fff640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 0x7fe94c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:50.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.275+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe9580f04e0 con 0x7fe9641034b0 2024-09-16T01:08:50.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.275+0000 7fe963fff640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 0x7fe94c0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe954009990 tx=0x7fe954009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:50.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.278+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe9580bcfc0 con 0x7fe9641034b0 2024-09-16T01:08:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.479+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe964101d20 con 0x7fe9641034b0 2024-09-16T01:08:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.479+0000 7fe9617fa640 1 -- 172.21.15.78:0/3527543499 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fe964101d20 con 0x7fe9641034b0 2024-09-16T01:08:50.481 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-16T01:08:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.481+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 msgr2=0x7fe94c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.481+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 0x7fe94c0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe954009990 tx=0x7fe954009110 comp rx=0 tx=0).stop 2024-09-16T01:08:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.481+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 msgr2=0x7fe964100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.481+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe958006010 tx=0x7fe9580042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 shutdown_connections 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe94c0a3240 0x7fe94c0a5700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe96410bfc0 0x7fe964101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9641034b0 0x7fe964100d30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 --2- 172.21.15.78:0/3527543499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe964102b90 0x7fe9641007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 >> 172.21.15.78:0/3527543499 conn(0x7fe9640fc630 msgr2=0x7fe9640fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 shutdown_connections 2024-09-16T01:08:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:50.482+0000 7fe96979d640 1 -- 172.21.15.78:0/3527543499 wait complete. 2024-09-16T01:08:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:51 smithi193 ceph-mon[35701]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:51 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3527543499' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:08:51.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:51 smithi078 ceph-mon[30937]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:51.316 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:51 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3527543499' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:08:51.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:51 smithi148 ceph-mon[35030]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:51.339 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:51 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3527543499' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:08:51.348 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-16T01:08:51.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:51.349 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-09-16T01:08:51.366 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:08:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:08:51.366 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:08:51.367 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-09-16T01:08:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:08:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42294sec preferred_lft 42294sec 2024-09-16T01:08:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link noprefixroute 2024-09-16T01:08:51.368 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:08:51.368 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T01:08:51.369 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:08:51.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:08:51.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:51.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:51.576 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:08:51.600 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-09-16T01:08:51.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:51.602 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-09-16T01:08:51.668 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-09-16T01:08:51.669 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:08:51.669 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:08:51.669 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:08:51.669 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:08:51.672 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:08:51.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:08:51.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:51.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:51.970 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-09-16T01:08:51.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:51.987 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-09-16T01:08:52.043 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:08:52.043 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:08:52.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:08:52.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-16T01:08:52.043 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:08:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:08:52.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:52.163 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-16T01:08:52.466 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:08:52.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:08:52.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:52.591 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:08:52.591 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:08:52.591 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-09-16T01:08:52.591 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T01:08:52.591 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:08:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:08:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:52.669 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-16T01:08:52.669 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:08:52.669 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:52.697 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-16T01:08:52.698 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:08:52.698 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:52.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-09-16T01:08:52.820 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T01:08:52.877 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-09-16T01:08:52.877 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-09-16T01:08:52.878 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-16T01:08:52.879 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/faf224185769657f548fb7f12a38df559f22e5e6bd823eb076ab995ab4255cb3/merged 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d51e8b5f2280cb08c0d28c39b8cba5c3dc7281052b738b56724c980d988ab18b/merged 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/83cc7f661207538b83bd8216d642c033fdfe1651cc12453d7f780040f811c117/merged 2024-09-16T01:08:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d1fc6a39a0b58c2ed3ac872c925bc20c52da30c65dc6ff0b4b84496dcd13e5df/merged 2024-09-16T01:08:52.881 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/0a9050303b286c2e7df6173c7706f2bc471f6718fdfcc1b2e1b31f365b3c4738/merged 2024-09-16T01:08:52.881 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/3b9eab345a42e0658b3bca326d30253e45357f0d67ed7261194df426f2be587f/merged 2024-09-16T01:08:52.881 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/ce283374115e9f5a381e2f3c9186f5826b99c10a1a73ffcb053ef710a024390b/merged 2024-09-16T01:08:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d28dc2c1455424defddd0df618322ccf95e539dab73c638edab5f4d1006749c5/merged 2024-09-16T01:08:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/3e317798cc61dc2b075de82168bc1413862c74082d0d256d1cc3810a861840ee/merged 2024-09-16T01:08:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/c595e181cf41486133500820b09980ae1dee098b7e0e3e432458e672fd517173/merged 2024-09-16T01:08:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-09-16T01:08:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:08:52.883 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:08:52.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:52.883 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-09-16T01:08:52.934 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T01:08:52.934 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-16T01:08:52.977 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-09-16T01:08:53.002 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-09-16T01:08:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-09-16T01:08:53.059 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/faf224185769657f548fb7f12a38df559f22e5e6bd823eb076ab995ab4255cb3/merged 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d51e8b5f2280cb08c0d28c39b8cba5c3dc7281052b738b56724c980d988ab18b/merged 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/0 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/83cc7f661207538b83bd8216d642c033fdfe1651cc12453d7f780040f811c117/merged 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d1fc6a39a0b58c2ed3ac872c925bc20c52da30c65dc6ff0b4b84496dcd13e5df/merged 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/0a9050303b286c2e7df6173c7706f2bc471f6718fdfcc1b2e1b31f365b3c4738/merged 2024-09-16T01:08:53.060 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/3b9eab345a42e0658b3bca326d30253e45357f0d67ed7261194df426f2be587f/merged 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/ce283374115e9f5a381e2f3c9186f5826b99c10a1a73ffcb053ef710a024390b/merged 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/d28dc2c1455424defddd0df618322ccf95e539dab73c638edab5f4d1006749c5/merged 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/3e317798cc61dc2b075de82168bc1413862c74082d0d256d1cc3810a861840ee/merged 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/containers/storage/overlay/c595e181cf41486133500820b09980ae1dee098b7e0e3e432458e672fd517173/merged 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:08:53.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:53.061 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-09-16T01:08:53.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2024-09-16T01:08:53.099+0000 7feb04b11480 -1 init, newargv = 0x561e3cfdabe0 newargc=15 2024-09-16T01:08:53.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[75151]: starting ceph client 2024-09-16T01:08:53.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[75151]: starting fuse 2024-09-16T01:08:53.114 INFO:teuthology.orchestra.run.smithi078.stdout:120 2024-09-16T01:08:53.115 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-16T01:08:53.115 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-09-16T01:08:53.116 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-09-16T01:08:53.117 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-09-16T01:08:53.117 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T01:08:53.117 DEBUG:teuthology.orchestra.run.smithi078:> 2024-09-16T01:08:53.117 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2024-09-16T01:08:53.117 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-09-16T01:08:53.176 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:53 smithi078 ceph-mon[30937]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:53.232 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.75151.asok 2024-09-16T01:08:53.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:53.242 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75151.asok 2024-09-16T01:08:53.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:53.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75151.asok status 2024-09-16T01:08:53.376 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "75151", 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-09-16T01:08:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 14718, 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 14718 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4241453742 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:08:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4241453742 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.14718 192.168.0.1:0/4241453742", 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/4241453742", 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 16, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 89, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-09-16T01:08:53.379 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:08:53.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:53.385 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-09-16T01:08:53.385 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:53 smithi148 ceph-mon[35030]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:53.401 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-09-16T01:08:53.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:53.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:53.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:53.468 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-16T01:08:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:53 smithi193 ceph-mon[35701]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:08:53.596 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2024-09-16T01:08:53.596 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-16T01:08:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:53.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:08:53.603 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:08:53.603 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:08:53.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T01:08:53.906 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 -- 172.21.15.78:0/1948115431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c106f10 msgr2=0x7f5f3c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 --2- 172.21.15.78:0/1948115431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c106f10 0x7f5f3c10ea90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009ca0 tx=0x7f5f3402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 -- 172.21.15.78:0/1948115431 shutdown_connections 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 --2- 172.21.15.78:0/1948115431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c106f10 0x7f5f3c10ea90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 --2- 172.21.15.78:0/1948115431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 --2- 172.21.15.78:0/1948115431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c101de0 0x7f5f3c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 -- 172.21.15.78:0/1948115431 >> 172.21.15.78:0/1948115431 conn(0x7f5f3c0fd8c0 msgr2=0x7f5f3c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 -- 172.21.15.78:0/1948115431 shutdown_connections 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.394+0000 7f5f41545640 1 -- 172.21.15.78:0/1948115431 wait complete. 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 Processor -- start 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 -- start start 2024-09-16T01:08:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c106f10 0x7f5f3c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c112010 con 0x7f5f3c101de0 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c111e90 con 0x7f5f3c102790 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.395+0000 7f5f41545640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c112190 con 0x7f5f3c106f10 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43610/0 (socket says 172.21.15.78:43610) 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 -- 172.21.15.78:0/2342908605 learned_addr learned my addr 172.21.15.78:0/2342908605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3b7fe640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c106f10 0x7f5f3c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f327fc640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 -- 172.21.15.78:0/2342908605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c106f10 msgr2=0x7f5f3c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c106f10 0x7f5f3c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 -- 172.21.15.78:0/2342908605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 msgr2=0x7f5f3c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 -- 172.21.15.78:0/2342908605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c19d520 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f327fc640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f3affd640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b4a0 tx=0x7f5f2c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c014070 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c19d810 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2c010af0 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c017d10 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.396+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c1a50f0 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.397+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c102240 con 0x7f5f3c101de0 2024-09-16T01:08:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.398+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5f2c017070 con 0x7f5f3c101de0 2024-09-16T01:08:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.399+0000 7f5f38ff9640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 0x7f5f280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.399+0000 7f5f327fc640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 0x7f5f280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.399+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5f2c0cf040 con 0x7f5f3c101de0 2024-09-16T01:08:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.399+0000 7f5f327fc640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 0x7f5f280a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f24006fd0 tx=0x7f5f24008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.403+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5f2c09bb20 con 0x7f5f3c101de0 2024-09-16T01:08:54.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.582+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5f3c1998d0 con 0x7f5f3c101de0 2024-09-16T01:08:54.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.583+0000 7f5f38ff9640 1 -- 172.21.15.78:0/2342908605 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f5f2c09b420 con 0x7f5f3c101de0 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/238352864 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4128193457 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2021750195 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3690753154 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1014137394 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:54.584 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/65187856 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:54.585 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-09-16T01:08:54.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 msgr2=0x7f5f280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 0x7f5f280a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f24006fd0 tx=0x7f5f24008040 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 msgr2=0x7f5f3c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b4a0 tx=0x7f5f2c00b960 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 shutdown_connections 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5f280a3240 0x7f5f280a5700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f3c106f10 0x7f5f3c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f3c102790 0x7f5f3c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 --2- 172.21.15.78:0/2342908605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c101de0 0x7f5f3c198570 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 >> 172.21.15.78:0/2342908605 conn(0x7f5f3c0fd8c0 msgr2=0x7f5f3c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 shutdown_connections 2024-09-16T01:08:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:54.585+0000 7f5f41545640 1 -- 172.21.15.78:0/2342908605 wait complete. 2024-09-16T01:08:55.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:55 smithi078 ceph-mon[30937]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:55.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:55 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2342908605' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:55.326 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:08:55.326 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:08:55.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T01:08:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:55 smithi148 ceph-mon[35030]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:55 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2342908605' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:55 smithi193 ceph-mon[35701]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:55 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2342908605' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:55.635 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 -- 172.21.15.78:0/178362883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1036e0 msgr2=0x7f584c107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 --2- 172.21.15.78:0/178362883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1036e0 0x7f584c107b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5834009930 tx=0x7f583402a420 comp rx=0 tx=0).stop 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 -- 172.21.15.78:0/178362883 shutdown_connections 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 --2- 172.21.15.78:0/178362883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1082a0 0x7f584c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 --2- 172.21.15.78:0/178362883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1036e0 0x7f584c107b90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 --2- 172.21.15.78:0/178362883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 0x7f584c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 -- 172.21.15.78:0/178362883 >> 172.21.15.78:0/178362883 conn(0x7f584c0fc7d0 msgr2=0x7f584c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 -- 172.21.15.78:0/178362883 shutdown_connections 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.128+0000 7f5850b51640 1 -- 172.21.15.78:0/178362883 wait complete. 2024-09-16T01:08:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 Processor -- start 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 -- start start 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 0x7f584c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1036e0 0x7f584c198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f584c1120e0 con 0x7f584c1082a0 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f584c111f60 con 0x7f584c1036e0 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5850b51640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f584c112260 con 0x7f584c102d30 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f584ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f584ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43640/0 (socket says 172.21.15.78:43640) 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f584ad9d640 1 -- 172.21.15.78:0/4077692216 learned_addr learned my addr 172.21.15.78:0/4077692216 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f584a59c640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 0x7f584c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f5843fff640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1036e0 0x7f584c198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.129+0000 7f584ad9d640 1 -- 172.21.15.78:0/4077692216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 msgr2=0x7f584c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f584ad9d640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 0x7f584c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f584ad9d640 1 -- 172.21.15.78:0/4077692216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1036e0 msgr2=0x7f584c198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f584ad9d640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1036e0 0x7f584c198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f584ad9d640 1 -- 172.21.15.78:0/4077692216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c19d5f0 con 0x7f584c1082a0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f584ad9d640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f584400b970 tx=0x7f584400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844019070 con 0x7f584c1082a0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f584400cb40 con 0x7f584c1082a0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c19d8e0 con 0x7f584c1082a0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844007650 con 0x7f584c1082a0 2024-09-16T01:08:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.130+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c1a51c0 con 0x7f584c1082a0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.131+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c103110 con 0x7f584c1082a0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.132+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f584401c070 con 0x7f584c1082a0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.133+0000 7f58437fe640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 0x7f58380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.133+0000 7f584a59c640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 0x7f58380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.133+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f58440cf5f0 con 0x7f584c1082a0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.134+0000 7f584a59c640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 0x7f58380a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f583c005fd0 tx=0x7f583c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.137+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f584409ba80 con 0x7f584c1082a0 2024-09-16T01:08:56.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.313+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f584c10a2b0 con 0x7f584c1082a0 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.314+0000 7f58437fe640 1 -- 172.21.15.78:0/4077692216 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+589 (secure 0 0 0) 0x7f584409fd80 con 0x7f584c1082a0 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/238352864 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4128193457 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:56.315 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2021750195 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3690753154 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1014137394 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/65187856 2024-09-17T01:01:12.814641+0000 2024-09-16T01:08:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 msgr2=0x7f58380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 0x7f58380a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f583c005fd0 tx=0x7f583c00b390 comp rx=0 tx=0).stop 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 msgr2=0x7f584c19cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f584400b970 tx=0x7f584400be30 comp rx=0 tx=0).stop 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 shutdown_connections 2024-09-16T01:08:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f58380a3240 0x7f58380a5700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584c1082a0 0x7f584c19cf10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f584c1036e0 0x7f584c198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 --2- 172.21.15.78:0/4077692216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f584c102d30 0x7f584c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 >> 172.21.15.78:0/4077692216 conn(0x7f584c0fc7d0 msgr2=0x7f584c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 shutdown_connections 2024-09-16T01:08:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:56.316+0000 7f5850b51640 1 -- 172.21.15.78:0/4077692216 wait complete. 2024-09-16T01:08:57.020 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-09-16T01:08:57.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:57.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-16T01:08:57.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[75151]: fuse finished with error 0 and tester_r 0 2024-09-16T01:08:57.048 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:08:57.078 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:08:57.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:08:57.078 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-16T01:08:57.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:08:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:57 smithi078 ceph-mon[30937]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:57 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4077692216' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:57 smithi148 ceph-mon[35030]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:57 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4077692216' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:57.424 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:57 smithi193 ceph-mon[35701]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:57 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4077692216' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.935+0000 7f55460d2640 1 -- 172.21.15.78:0/271698862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 msgr2=0x7f55401021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.935+0000 7f55460d2640 1 --2- 172.21.15.78:0/271698862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401021c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5530009900 tx=0x7f553002a410 comp rx=0 tx=0).stop 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 -- 172.21.15.78:0/271698862 shutdown_connections 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 --2- 172.21.15.78:0/271698862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 0x7f554010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 --2- 172.21.15.78:0/271698862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 --2- 172.21.15.78:0/271698862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401021c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 -- 172.21.15.78:0/271698862 >> 172.21.15.78:0/271698862 conn(0x7f55400fd8c0 msgr2=0x7f55400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 -- 172.21.15.78:0/271698862 shutdown_connections 2024-09-16T01:08:57.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 -- 172.21.15.78:0/271698862 wait complete. 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 Processor -- start 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 -- start start 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.936+0000 7f55460d2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 0x7f554019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5540111f90 con 0x7f5540101de0 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5540111e10 con 0x7f5540102790 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5540112110 con 0x7f5540106f10 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43660/0 (socket says 172.21.15.78:43660) 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 -- 172.21.15.78:0/3447743452 learned_addr learned my addr 172.21.15.78:0/3447743452 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f5536ffd640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:57.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553ffff640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 0x7f554019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 -- 172.21.15.78:0/3447743452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 msgr2=0x7f554019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 0x7f554019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 -- 172.21.15.78:0/3447743452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 msgr2=0x7f55401989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 -- 172.21.15.78:0/3447743452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554019d460 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f5536ffd640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553f7fe640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5530005ab0 tx=0x7f55300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553003f030 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554019d6f0 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5530003ce0 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530035070 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.937+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55401a5030 con 0x7f5540101de0 2024-09-16T01:08:57.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.938+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55401032c0 con 0x7f5540101de0 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.940+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5530042030 con 0x7f5540101de0 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.940+0000 7f553d7fa640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 0x7f552c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.941+0000 7f5536ffd640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 0x7f552c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.941+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f55300f0740 con 0x7f5540101de0 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.941+0000 7f5536ffd640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 0x7f552c0a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55401087d0 tx=0x7f5528008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:57.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:57.943+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f55300bd220 con 0x7f5540101de0 2024-09-16T01:08:58.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.116+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55401021c0 con 0x7f5540101de0 2024-09-16T01:08:58.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.117+0000 7f553d7fa640 1 -- 172.21.15.78:0/3447743452 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7f55300bcb20 con 0x7f5540101de0 2024-09-16T01:08:58.118 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:58.118 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 msgr2=0x7f552c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 0x7f552c0a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55401087d0 tx=0x7f5528008040 comp rx=0 tx=0).stop 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 msgr2=0x7f55401984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5530005ab0 tx=0x7f55300042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 shutdown_connections 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f552c0a3240 0x7f552c0a5700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5540106f10 0x7f554019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5540102790 0x7f55401989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 --2- 172.21.15.78:0/3447743452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5540101de0 0x7f55401984b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 >> 172.21.15.78:0/3447743452 conn(0x7f55400fd8c0 msgr2=0x7f55400feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 shutdown_connections 2024-09-16T01:08:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:58.120+0000 7f55460d2640 1 -- 172.21.15.78:0/3447743452 wait complete. 2024-09-16T01:08:58.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:08:59.145 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:08:59.266 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:59 smithi078 ceph-mon[30937]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:59.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:08:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3447743452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:59 smithi148 ceph-mon[35030]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:08:59 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3447743452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:59.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:59 smithi193 ceph-mon[35701]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:08:59.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:08:59 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3447743452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:08:59.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 -- 172.21.15.78:0/1342581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 msgr2=0x7f80d01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:59.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 --2- 172.21.15.78:0/1342581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01021c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009900 tx=0x7f80c402a410 comp rx=0 tx=0).stop 2024-09-16T01:08:59.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 -- 172.21.15.78:0/1342581024 shutdown_connections 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 --2- 172.21.15.78:0/1342581024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 --2- 172.21.15.78:0/1342581024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 0x7f80d01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 --2- 172.21.15.78:0/1342581024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01021c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 -- 172.21.15.78:0/1342581024 >> 172.21.15.78:0/1342581024 conn(0x7f80d00fd880 msgr2=0x7f80d00ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 -- 172.21.15.78:0/1342581024 shutdown_connections 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.648+0000 7f80d785f640 1 -- 172.21.15.78:0/1342581024 wait complete. 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 Processor -- start 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 -- start start 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 0x7f80d01989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0111f90 con 0x7f80d0101de0 2024-09-16T01:08:59.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0111e10 con 0x7f80d0102790 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d785f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0112110 con 0x7f80d0106f10 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43686/0 (socket says 172.21.15.78:43686) 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 -- 172.21.15.78:0/640039576 learned_addr learned my addr 172.21.15.78:0/640039576 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d5dfc640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 -- 172.21.15.78:0/640039576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 msgr2=0x7f80d019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d4dfa640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 0x7f80d01989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 -- 172.21.15.78:0/640039576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 msgr2=0x7f80d01989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 0x7f80d01989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d55fb640 1 -- 172.21.15.78:0/640039576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d019d450 con 0x7f80d0101de0 2024-09-16T01:08:59.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.649+0000 7f80d5dfc640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80d55fb640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f80c4005ab0 tx=0x7f80c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c403f030 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c4003ce0 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d019d6e0 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4035070 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.650+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d01a5020 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.651+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d01032c0 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.652+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f80c4042030 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.653+0000 7f80c27fc640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.653+0000 7f80d4dfa640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.653+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f80c40f04f0 con 0x7f80d0101de0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.653+0000 7f80d4dfa640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 0x7f80bc0a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f80d00ffe40 tx=0x7f80cc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:08:59.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.657+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80c40bcfd0 con 0x7f80d0101de0 2024-09-16T01:08:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.830+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80d010a0f0 con 0x7f80d0101de0 2024-09-16T01:08:59.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.831+0000 7f80c27fc640 1 -- 172.21.15.78:0/640039576 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7f80c40bc8d0 con 0x7f80d0101de0 2024-09-16T01:08:59.832 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:08:59.833 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 msgr2=0x7f80bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 0x7f80bc0a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f80d00ffe40 tx=0x7f80cc008040 comp rx=0 tx=0).stop 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 msgr2=0x7f80d01984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f80c4005ab0 tx=0x7f80c40042f0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 shutdown_connections 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80d0106f10 0x7f80d019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d0102790 0x7f80d01989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 --2- 172.21.15.78:0/640039576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80d0101de0 0x7f80d01984a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:08:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 >> 172.21.15.78:0/640039576 conn(0x7f80d00fd880 msgr2=0x7f80d00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:08:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 shutdown_connections 2024-09-16T01:08:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:08:59.834+0000 7f80d785f640 1 -- 172.21.15.78:0/640039576 wait complete. 2024-09-16T01:09:00.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:00 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/640039576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:00 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/640039576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:00.430 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:00 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/640039576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:00.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:09:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:01.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:01 smithi078 ceph-mon[30937]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:01 smithi193 ceph-mon[35701]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:01.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:01 smithi148 ceph-mon[35030]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:01.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.401+0000 7f9d20b29640 1 -- 172.21.15.78:0/3875400332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c101df0 msgr2=0x7f9d1c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.401+0000 7f9d20b29640 1 --2- 172.21.15.78:0/3875400332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1021d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009ca0 tx=0x7f9d0c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 -- 172.21.15.78:0/3875400332 shutdown_connections 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 --2- 172.21.15.78:0/3875400332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c106f20 0x7f9d1c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 --2- 172.21.15.78:0/3875400332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 --2- 172.21.15.78:0/3875400332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1021d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 -- 172.21.15.78:0/3875400332 >> 172.21.15.78:0/3875400332 conn(0x7f9d1c0fd8b0 msgr2=0x7f9d1c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 -- 172.21.15.78:0/3875400332 shutdown_connections 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 -- 172.21.15.78:0/3875400332 wait complete. 2024-09-16T01:09:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 Processor -- start 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.402+0000 7f9d20b29640 1 -- start start 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c111f40 con 0x7f9d1c106f20 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c111dc0 con 0x7f9d1c1027a0 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1120c0 con 0x7f9d1c101df0 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43702/0 (socket says 172.21.15.78:43702) 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 -- 172.21.15.78:0/2590641557 learned_addr learned my addr 172.21.15.78:0/2590641557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d19d9b640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:01.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1a59c640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 -- 172.21.15.78:0/2590641557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c101df0 msgr2=0x7f9d1c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 -- 172.21.15.78:0/2590641557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 msgr2=0x7f9d1c198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 -- 172.21.15.78:0/2590641557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c19d390 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d19d9b640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d1ad9d640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400c970 tx=0x7f9d1400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14007bb0 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c19d680 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d14007d50 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400f450 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.403+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c1a4f60 con 0x7f9d1c106f20 2024-09-16T01:09:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.404+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1c1021d0 con 0x7f9d1c106f20 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.405+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9d14004f60 con 0x7f9d1c106f20 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.406+0000 7f9d137fe640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 0x7f9d080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.406+0000 7f9d1a59c640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 0x7f9d080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.406+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9d140cf910 con 0x7f9d1c106f20 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.407+0000 7f9d1a59c640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 0x7f9d080a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c02f540 tx=0x7f9d0c02f310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.410+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9d14014030 con 0x7f9d1c106f20 2024-09-16T01:09:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:01.611+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9d1c109b70 con 0x7f9d1c106f20 2024-09-16T01:09:02.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.072+0000 7f9d137fe640 1 -- 172.21.15.78:0/2590641557 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f9d1409c390 con 0x7f9d1c106f20 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 msgr2=0x7f9d080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 0x7f9d080a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c02f540 tx=0x7f9d0c02f310 comp rx=0 tx=0).stop 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 msgr2=0x7f9d1c19ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9d1400c970 tx=0x7f9d1400ce30 comp rx=0 tx=0).stop 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 shutdown_connections 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9d080a3240 0x7f9d080a5700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1c106f20 0x7f9d1c19ccb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d1c1027a0 0x7f9d1c198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:02.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 --2- 172.21.15.78:0/2590641557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d1c101df0 0x7f9d1c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:02.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 >> 172.21.15.78:0/2590641557 conn(0x7f9d1c0fd8b0 msgr2=0x7f9d1c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:02.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 shutdown_connections 2024-09-16T01:09:02.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:02.074+0000 7f9d20b29640 1 -- 172.21.15.78:0/2590641557 wait complete. 2024-09-16T01:09:02.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:02 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T01:09:02.396 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:02 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T01:09:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:02 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-16T01:09:02.854 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-09-16T01:09:02.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-09-16T01:09:03.106 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:03 smithi148 ceph-mon[35030]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:03.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:03 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T01:09:03.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:03 smithi148 ceph-mon[35030]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:03.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:03 smithi078 ceph-mon[30937]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:03.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:03 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T01:09:03.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:03 smithi078 ceph-mon[30937]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:03 smithi193 ceph-mon[35701]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:03 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2590641557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-16T01:09:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:03 smithi193 ceph-mon[35701]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 -- 172.21.15.78:0/1519368269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102ef0 msgr2=0x7fded410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:03.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 --2- 172.21.15.78:0/1519368269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102ef0 0x7fded410da40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009ca0 tx=0x7fdecc02a950 comp rx=0 tx=0).stop 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 -- 172.21.15.78:0/1519368269 shutdown_connections 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 --2- 172.21.15.78:0/1519368269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102ef0 0x7fded410da40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 --2- 172.21.15.78:0/1519368269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4102570 0x7fded41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 --2- 172.21.15.78:0/1519368269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4100e20 0x7fded4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 -- 172.21.15.78:0/1519368269 >> 172.21.15.78:0/1519368269 conn(0x7fded40fc7d0 msgr2=0x7fded40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 -- 172.21.15.78:0/1519368269 shutdown_connections 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 -- 172.21.15.78:0/1519368269 wait complete. 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.635+0000 7fdedb243640 1 Processor -- start 2024-09-16T01:09:03.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 -- start start 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 0x7fded410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4102ef0 0x7fded410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fded41120a0 con 0x7fded4102570 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fded4111f20 con 0x7fded4100e20 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fdedb243640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fded4112220 con 0x7fded4102ef0 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43714/0 (socket says 172.21.15.78:43714) 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 -- 172.21.15.78:0/4073825843 learned_addr learned my addr 172.21.15.78:0/4073825843 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded97e0640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4102ef0 0x7fded410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded8fdf640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 0x7fded410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 -- 172.21.15.78:0/4073825843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4102ef0 msgr2=0x7fded410ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4102ef0 0x7fded410ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 -- 172.21.15.78:0/4073825843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 msgr2=0x7fded410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 0x7fded410e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 -- 172.21.15.78:0/4073825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded41a68f0 con 0x7fded4102570 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded8fdf640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 0x7fded410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded3fff640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdec000b4a0 tx=0x7fdec000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.636+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec0014070 con 0x7fded4102570 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.637+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec0010af0 con 0x7fded4102570 2024-09-16T01:09:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.637+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded41a6be0 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.637+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec0017d10 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.637+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded41a7120 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.638+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded4101200 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.639+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdec0017070 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.639+0000 7fded1ffb640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 0x7fdec40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.640+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdec00ceec0 con 0x7fded4102570 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.640+0000 7fded8fdf640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 0x7fdec40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.640+0000 7fded8fdf640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 0x7fdec40a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdec800baa0 tx=0x7fdec8005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:03.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.644+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdec009b9a0 con 0x7fded4102570 2024-09-16T01:09:03.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:03.847+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fded41a72f0 con 0x7fded4102570 2024-09-16T01:09:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.079+0000 7fded1ffb640 1 -- 172.21.15.78:0/4073825843 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fdec009b2a0 con 0x7fded4102570 2024-09-16T01:09:04.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 msgr2=0x7fdec40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:04.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 0x7fdec40a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdec800baa0 tx=0x7fdec8005d90 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 msgr2=0x7fded410ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdec000b4a0 tx=0x7fdec000b960 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 shutdown_connections 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdec40a3240 0x7fdec40a5700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fded4102ef0 0x7fded410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fded4102570 0x7fded410ea10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 --2- 172.21.15.78:0/4073825843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded4100e20 0x7fded410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 >> 172.21.15.78:0/4073825843 conn(0x7fded40fc7d0 msgr2=0x7fded4106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 shutdown_connections 2024-09-16T01:09:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:04.081+0000 7fdedb243640 1 -- 172.21.15.78:0/4073825843 wait complete. 2024-09-16T01:09:04.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T01:09:04.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:04 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T01:09:04.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:04 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-16T01:09:04.765 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-16T01:09:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:05.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T01:09:05.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:05 smithi078 ceph-mon[30937]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T01:09:05.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:05.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T01:09:05.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T01:09:05.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T01:09:05.390 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:05.390 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T01:09:05.390 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:05 smithi148 ceph-mon[35030]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T01:09:05.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 -- 172.21.15.78:0/2928928118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba0103660 msgr2=0x7f0ba010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 --2- 172.21.15.78:0/2928928118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba0103660 0x7f0ba010bb50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009900 tx=0x7f0b9402a480 comp rx=0 tx=0).stop 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 -- 172.21.15.78:0/2928928118 shutdown_connections 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 --2- 172.21.15.78:0/2928928118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba010c170 0x7f0ba010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 --2- 172.21.15.78:0/2928928118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba0103660 0x7f0ba010bb50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 --2- 172.21.15.78:0/2928928118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 -- 172.21.15.78:0/2928928118 >> 172.21.15.78:0/2928928118 conn(0x7f0ba00fc7a0 msgr2=0x7f0ba00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 -- 172.21.15.78:0/2928928118 shutdown_connections 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.509+0000 7f0ba6402640 1 -- 172.21.15.78:0/2928928118 wait complete. 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 Processor -- start 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 -- start start 2024-09-16T01:09:05.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba0103660 0x7f0ba0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0111f70 con 0x7f0ba010c170 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba0111df0 con 0x7f0ba0102d40 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba6402640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba01120f0 con 0x7f0ba0103660 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35702/0 (socket says 172.21.15.78:35702) 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 -- 172.21.15.78:0/44380612 learned_addr learned my addr 172.21.15.78:0/44380612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0b9f7fe640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba0103660 0x7f0ba0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0b9ffff640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 -- 172.21.15.78:0/44380612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba0103660 msgr2=0x7f0ba0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:05.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba0103660 0x7f0ba0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 -- 172.21.15.78:0/44380612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 msgr2=0x7f0ba01009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba01009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0ba499f640 1 -- 172.21.15.78:0/44380612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba01a68b0 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.510+0000 7f0b9ffff640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0ba499f640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800cce0 tx=0x7f0b9800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9800f450 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b9800fad0 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba01a6ba0 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98018c10 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.511+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba01a7090 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.512+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba0103120 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.514+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0b9800f5f0 con 0x7f0ba010c170 2024-09-16T01:09:05.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.514+0000 7f0b9d7fa640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:05.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.515+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0b980d01f0 con 0x7f0ba010c170 2024-09-16T01:09:05.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.515+0000 7f0b9ffff640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:05.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.515+0000 7f0b9ffff640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 0x7f0b880a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0b900058c0 tx=0x7f0b90005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:05.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.518+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0b98014030 con 0x7f0ba010c170 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4073825843' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-16T01:09:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:05 smithi193 ceph-mon[35701]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-16T01:09:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:05.718+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f0ba01071e0 con 0x7f0ba010c170 2024-09-16T01:09:06.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.100+0000 7f0b9d7fa640 1 -- 172.21.15.78:0/44380612 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f0b9809ce00 con 0x7f0ba010c170 2024-09-16T01:09:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 msgr2=0x7f0b880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 0x7f0b880a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0b900058c0 tx=0x7f0b90005d30 comp rx=0 tx=0).stop 2024-09-16T01:09:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 msgr2=0x7f0ba0101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800cce0 tx=0x7f0b9800a620 comp rx=0 tx=0).stop 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 shutdown_connections 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b880a3240 0x7f0b880a5700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ba010c170 0x7f0ba0101420 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ba0103660 0x7f0ba0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 --2- 172.21.15.78:0/44380612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ba0102d40 0x7f0ba01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.102+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 >> 172.21.15.78:0/44380612 conn(0x7f0ba00fc7a0 msgr2=0x7f0ba00fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.103+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 shutdown_connections 2024-09-16T01:09:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:06.103+0000 7f0ba6402640 1 -- 172.21.15.78:0/44380612 wait complete. 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:active 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: Cluster is now healthy 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-09-16T01:09:06.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:06 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T01:09:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:active 2024-09-16T01:09:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T01:09:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T01:09:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: Cluster is now healthy 2024-09-16T01:09:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T01:09:06.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-09-16T01:09:06.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-09-16T01:09:06.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:06 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] up:active 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: Cluster is now healthy 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-09-16T01:09:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:06 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-16T01:09:06.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-16T01:09:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:07 smithi078 ceph-mon[30937]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:07 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T01:09:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:07 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:active 2024-09-16T01:09:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:07 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-09-16T01:09:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:07 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:07 smithi148 ceph-mon[35030]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:07 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T01:09:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:07 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:active 2024-09-16T01:09:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:07 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-09-16T01:09:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:07 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:07 smithi193 ceph-mon[35701]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-16T01:09:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:07 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/44380612' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-16T01:09:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:07 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] up:active 2024-09-16T01:09:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:07 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-09-16T01:09:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:07 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:07.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- 172.21.15.78:0/3388175972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18101df0 msgr2=0x7f3f181021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:07.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/3388175972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18101df0 0x7f3f181021d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009900 tx=0x7f3f0802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:07.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- 172.21.15.78:0/3388175972 shutdown_connections 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/3388175972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18106f20 0x7f3f1810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/3388175972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 0x7f3f181067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/3388175972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18101df0 0x7f3f181021d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- 172.21.15.78:0/3388175972 >> 172.21.15.78:0/3388175972 conn(0x7f3f180fd8f0 msgr2=0x7f3f180ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- 172.21.15.78:0/3388175972 shutdown_connections 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- 172.21.15.78:0/3388175972 wait complete. 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 Processor -- start 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.721+0000 7f3f1c85d640 1 -- start start 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 0x7f3f18198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 0x7f3f181989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:07.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18111fc0 con 0x7f3f18106f20 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18111e40 con 0x7f3f18101df0 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f18112140 con 0x7f3f181027a0 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35718/0 (socket says 172.21.15.78:35718) 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 -- 172.21.15.78:0/4051968190 learned_addr learned my addr 172.21.15.78:0/4051968190 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f15d9b640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 0x7f3f181989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1659c640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 0x7f3f18198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 -- 172.21.15.78:0/4051968190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 msgr2=0x7f3f181989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 0x7f3f181989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 -- 172.21.15.78:0/4051968190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 msgr2=0x7f3f18198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:07.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 0x7f3f18198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 -- 172.21.15.78:0/4051968190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1819d430 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1659c640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 0x7f3f18198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f16d9d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000b970 tx=0x7f3f1000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10019070 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1819d720 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f1000cb40 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.722+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10007650 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.723+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f181a5000 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.723+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f181021d0 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.724+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3f1001c070 con 0x7f3f18106f20 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.725+0000 7f3f077fe640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 0x7f3f000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.725+0000 7f3f1659c640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 0x7f3f000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:07.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.726+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f100cf140 con 0x7f3f18106f20 2024-09-16T01:09:07.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.726+0000 7f3f1659c640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 0x7f3f000a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f08005ab0 tx=0x7f3f08005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:07.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.730+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3f1009b5d0 con 0x7f3f18106f20 2024-09-16T01:09:07.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:07.934+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3f1802d7c0 con 0x7f3f18106f20 2024-09-16T01:09:08.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.106+0000 7f3f077fe640 1 -- 172.21.15.78:0/4051968190 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f3f1009f8d0 con 0x7f3f18106f20 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 msgr2=0x7f3f000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 0x7f3f000a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f08005ab0 tx=0x7f3f08005a20 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 msgr2=0x7f3f1819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f1000b970 tx=0x7f3f1000be30 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 shutdown_connections 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3f000a3240 0x7f3f000a5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f18106f20 0x7f3f1819cd50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f181027a0 0x7f3f181989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 --2- 172.21.15.78:0/4051968190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f18101df0 0x7f3f18198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 >> 172.21.15.78:0/4051968190 conn(0x7f3f180fd8f0 msgr2=0x7f3f180fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 shutdown_connections 2024-09-16T01:09:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:08.109+0000 7f3f1c85d640 1 -- 172.21.15.78:0/4051968190 wait complete. 2024-09-16T01:09:08.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T01:09:08.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:08 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T01:09:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:08 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-16T01:09:08.885 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:09:08 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: 2024-09-16T01:09:08.580+0000 7f4402b5b640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-16T01:09:08.922 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-09-16T01:09:08.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-16T01:09:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-mon[30937]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-09-16T01:09:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T01:09:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:09 smithi148 ceph-mon[35030]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-09-16T01:09:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:09 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T01:09:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:09 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:09 smithi193 ceph-mon[35701]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.3 KiB/s wr, 5 op/s 2024-09-16T01:09:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:09 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4051968190' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-16T01:09:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:09 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:09:09.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 -- 172.21.15.78:0/3571167664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498101420 msgr2=0x7f549810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:09.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 --2- 172.21.15.78:0/3571167664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498101420 0x7f549810d950 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5490009cd0 tx=0x7f549002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 -- 172.21.15.78:0/3571167664 shutdown_connections 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 --2- 172.21.15.78:0/3571167664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498101420 0x7f549810d950 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 --2- 172.21.15.78:0/3571167664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498100a80 0x7f5498100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 --2- 172.21.15.78:0/3571167664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498106a80 0x7f5498106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 -- 172.21.15.78:0/3571167664 >> 172.21.15.78:0/3571167664 conn(0x7f54980fc7a0 msgr2=0x7f54980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 -- 172.21.15.78:0/3571167664 shutdown_connections 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.706+0000 7f549f36d640 1 -- 172.21.15.78:0/3571167664 wait complete. 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 Processor -- start 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 -- start start 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498100a80 0x7f5498197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:09.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498101420 0x7f54981977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5498110e80 con 0x7f5498106a80 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5498110d00 con 0x7f5498100a80 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549f36d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5498111000 con 0x7f5498101420 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549d90a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549d90a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35728/0 (socket says 172.21.15.78:35728) 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549d90a640 1 -- 172.21.15.78:0/2367360638 learned_addr learned my addr 172.21.15.78:0/2367360638 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549c908640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498101420 0x7f54981977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.707+0000 7f549d109640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498100a80 0x7f5498197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 -- 172.21.15.78:0/2367360638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498101420 msgr2=0x7f54981977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498101420 0x7f54981977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 -- 172.21.15.78:0/2367360638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498100a80 msgr2=0x7f5498197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498100a80 0x7f5498197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 -- 172.21.15.78:0/2367360638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549819c230 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549d90a640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5490005cc0 tx=0x7f5490003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490004040 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54900046c0 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549819c4c0 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490046480 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.708+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54981a3e00 con 0x7f5498106a80 2024-09-16T01:09:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.709+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5498100ee0 con 0x7f5498106a80 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.710+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f54900305a0 con 0x7f5498106a80 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.711+0000 7f548e7fc640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 0x7f54840a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.711+0000 7f549d109640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 0x7f54840a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.711+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f549002e030 con 0x7f5498106a80 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.711+0000 7f549d109640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 0x7f54840a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f549400cac0 tx=0x7f549400c4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.715+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f549004b060 con 0x7f5498106a80 2024-09-16T01:09:09.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.904+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f54980fd480 con 0x7f54840a32c0 2024-09-16T01:09:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.929+0000 7f548e7fc640 1 -- 172.21.15.78:0/2367360638 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f54980fd480 con 0x7f54840a32c0 2024-09-16T01:09:09.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 msgr2=0x7f54840a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:09.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 0x7f54840a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f549400cac0 tx=0x7f549400c4a0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 msgr2=0x7f549819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5490005cc0 tx=0x7f5490003970 comp rx=0 tx=0).stop 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 shutdown_connections 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f54840a32c0 0x7f54840a5780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5498106a80 0x7f549819bb50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5498101420 0x7f54981977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 --2- 172.21.15.78:0/2367360638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5498100a80 0x7f5498197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 >> 172.21.15.78:0/2367360638 conn(0x7f54980fc7a0 msgr2=0x7f54980fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 shutdown_connections 2024-09-16T01:09:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:09.931+0000 7f549f36d640 1 -- 172.21.15.78:0/2367360638 wait complete. 2024-09-16T01:09:10.162 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:09:09.910+0000 7f3edbb79640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:09:10.162 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:09:09.910+0000 7f3edbb79640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:09:10.162 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:09:09.910+0000 7f3edbb79640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:09:10.162 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:09:09.910+0000 7f3edbb79640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:09:10.162 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:09:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:09:09.910+0000 7f3edbb79640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:09:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:10 smithi078 ceph-mon[30937]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:10 smithi078 ceph-mon[30937]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-09-16T01:09:10.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:10 smithi078 ceph-mon[30937]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T01:09:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:10 smithi193 ceph-mon[35701]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:10 smithi193 ceph-mon[35701]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-09-16T01:09:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:10 smithi193 ceph-mon[35701]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T01:09:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:10 smithi148 ceph-mon[35030]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:10 smithi148 ceph-mon[35030]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2024-09-16T01:09:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:10 smithi148 ceph-mon[35030]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-16T01:09:10.776 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-16T01:09:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:11.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.527+0000 7f5c4b9db640 1 -- 172.21.15.78:0/4257116803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44100a80 msgr2=0x7f5c44100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.527+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/4257116803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44100a80 0x7f5c44100ee0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009cd0 tx=0x7f5c3802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 -- 172.21.15.78:0/4257116803 shutdown_connections 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/4257116803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c4410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/4257116803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44100a80 0x7f5c44100ee0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/4257116803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44106a80 0x7f5c44106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 -- 172.21.15.78:0/4257116803 >> 172.21.15.78:0/4257116803 conn(0x7f5c440fc7a0 msgr2=0x7f5c440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 -- 172.21.15.78:0/4257116803 shutdown_connections 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 -- 172.21.15.78:0/4257116803 wait complete. 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 Processor -- start 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.528+0000 7f5c4b9db640 1 -- start start 2024-09-16T01:09:11.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44100a80 0x7f5c44197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c44197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44110e80 con 0x7f5c44106a80 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44110d00 con 0x7f5c44100a80 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c4b9db640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44111000 con 0x7f5c44101420 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35760/0 (socket says 172.21.15.78:35760) 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 -- 172.21.15.78:0/1138775172 learned_addr learned my addr 172.21.15.78:0/1138775172 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49777640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44100a80 0x7f5c44197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c48f76640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c44197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 -- 172.21.15.78:0/1138775172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 msgr2=0x7f5c44197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c44197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 -- 172.21.15.78:0/1138775172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44100a80 msgr2=0x7f5c44197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44100a80 0x7f5c44197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 -- 172.21.15.78:0/1138775172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4419c2f0 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c48f76640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c44197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c49f78640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c970 tx=0x7f5c3400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.529+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34007bb0 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.530+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c34007d50 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.530+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4419c5e0 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.530+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34018800 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.530+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c441a3ec0 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.530+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c44100ee0 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.532+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5c3400fa50 con 0x7f5c44106a80 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.532+0000 7f5c427fc640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 0x7f5c2c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.532+0000 7f5c49777640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 0x7f5c2c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.532+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5c340cfd30 con 0x7f5c44106a80 2024-09-16T01:09:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.533+0000 7f5c49777640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 0x7f5c2c0a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00bb60 tx=0x7f5c3c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.536+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5c340d4070 con 0x7f5c44106a80 2024-09-16T01:09:11.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.727+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5c44062630 con 0x7f5c2c0a32d0 2024-09-16T01:09:11.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.753+0000 7f5c427fc640 1 -- 172.21.15.78:0/1138775172 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5c44062630 con 0x7f5c2c0a32d0 2024-09-16T01:09:11.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 msgr2=0x7f5c2c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 0x7f5c2c0a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00bb60 tx=0x7f5c3c00b390 comp rx=0 tx=0).stop 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 msgr2=0x7f5c4419bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c970 tx=0x7f5c3400ce30 comp rx=0 tx=0).stop 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 shutdown_connections 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5c2c0a32d0 0x7f5c2c0a5790 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c44106a80 0x7f5c4419bc10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c44101420 0x7f5c44197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 --2- 172.21.15.78:0/1138775172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c44100a80 0x7f5c44197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 >> 172.21.15.78:0/1138775172 conn(0x7f5c440fc7a0 msgr2=0x7f5c440fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 shutdown_connections 2024-09-16T01:09:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:11.755+0000 7f5c4b9db640 1 -- 172.21.15.78:0/1138775172 wait complete. 2024-09-16T01:09:12.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-16T01:09:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:13.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:13 smithi078 ceph-mon[30937]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:13.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:13 smithi078 ceph-mon[30937]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/3184333008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 msgr2=0x7fe0c010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/3184333008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c010ec40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009cd0 tx=0x7fe0b802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/3184333008 shutdown_connections 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/3184333008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c010ec40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/3184333008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c01036e0 0x7fe0c0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/3184333008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c0102d30 0x7fe0c0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/3184333008 >> 172.21.15.78:0/3184333008 conn(0x7fe0c00fc7d0 msgr2=0x7fe0c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/3184333008 shutdown_connections 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/3184333008 wait complete. 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 Processor -- start 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 -- start start 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c0102d30 0x7fe0c0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.278+0000 7fe0c6bbe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 0x7fe0c0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c6bbe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c6bbe640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c01120a0 con 0x7fe0c01082a0 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c6bbe640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0111f20 con 0x7fe0c01036e0 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c6bbe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0112220 con 0x7fe0c0102d30 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:13.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35776/0 (socket says 172.21.15.78:35776) 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 -- 172.21.15.78:0/4144785384 learned_addr learned my addr 172.21.15.78:0/4144785384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0bffff640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 0x7fe0c0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c495a640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c0102d30 0x7fe0c0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 -- 172.21.15.78:0/4144785384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c0102d30 msgr2=0x7fe0c0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c0102d30 0x7fe0c0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 -- 172.21.15.78:0/4144785384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 msgr2=0x7fe0c0198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 0x7fe0c0198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 -- 172.21.15.78:0/4144785384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c019d4e0 con 0x7fe0c01082a0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0bffff640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 0x7fe0c0198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c515b640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8005cc0 tx=0x7fe0b8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8004040 con 0x7fe0c01082a0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c019d770 con 0x7fe0c01082a0 2024-09-16T01:09:13.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b80046c0 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.279+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8046480 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.280+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c01a50b0 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.280+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c0103110 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.281+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe0b80305a0 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.282+0000 7fe0bdffb640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 0x7fe0b00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.282+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe0b802e030 con 0x7fe0c01082a0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.282+0000 7fe0c495a640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 0x7fe0b00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.283+0000 7fe0c495a640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 0x7fe0b00a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4005fd0 tx=0x7fe0b4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:13.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.286+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe0b804b060 con 0x7fe0c01082a0 2024-09-16T01:09:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:13 smithi148 ceph-mon[35030]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:13 smithi148 ceph-mon[35030]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T01:09:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.481+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fe0c0062630 con 0x7fe0b00a32d0 2024-09-16T01:09:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.487+0000 7fe0bdffb640 1 -- 172.21.15.78:0/4144785384 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fe0c0062630 con 0x7fe0b00a32d0 2024-09-16T01:09:13.488 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_0/4b0e450c-537c-4af3-8b89-c4577e19252b 2024-09-16T01:09:13.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 msgr2=0x7fe0b00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:13.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 0x7fe0b00a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4005fd0 tx=0x7fe0b4005850 comp rx=0 tx=0).stop 2024-09-16T01:09:13.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 msgr2=0x7fe0c019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:13.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8005cc0 tx=0x7fe0b8003970 comp rx=0 tx=0).stop 2024-09-16T01:09:13.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 shutdown_connections 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0b00a32d0 0x7fe0b00a5790 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0c01082a0 0x7fe0c019ce00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0c01036e0 0x7fe0c0198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 --2- 172.21.15.78:0/4144785384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0c0102d30 0x7fe0c0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 >> 172.21.15.78:0/4144785384 conn(0x7fe0c00fc7d0 msgr2=0x7fe0c0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 shutdown_connections 2024-09-16T01:09:13.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:13.489+0000 7fe0c6bbe640 1 -- 172.21.15.78:0/4144785384 wait complete. 2024-09-16T01:09:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:13 smithi193 ceph-mon[35701]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:13 smithi193 ceph-mon[35701]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-16T01:09:14.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-16T01:09:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 -- 172.21.15.78:0/360303724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8106a80 msgr2=0x7ff4f8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 --2- 172.21.15.78:0/360303724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8106a80 0x7ff4f8106e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4009930 tx=0x7ff4f402a420 comp rx=0 tx=0).stop 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 -- 172.21.15.78:0/360303724 shutdown_connections 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 --2- 172.21.15.78:0/360303724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8101420 0x7ff4f810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 --2- 172.21.15.78:0/360303724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 0x7ff4f8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 --2- 172.21.15.78:0/360303724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8106a80 0x7ff4f8106e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 -- 172.21.15.78:0/360303724 >> 172.21.15.78:0/360303724 conn(0x7ff4f80fc7a0 msgr2=0x7ff4f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 -- 172.21.15.78:0/360303724 shutdown_connections 2024-09-16T01:09:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.024+0000 7ff5005c8640 1 -- 172.21.15.78:0/360303724 wait complete. 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 Processor -- start 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 -- start start 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 0x7ff4f8198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8106a80 0x7ff4f819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8110f20 con 0x7ff4f8101420 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8110da0 con 0x7ff4f8100a80 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff5005c8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f81110a0 con 0x7ff4f8106a80 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50156/0 (socket says 172.21.15.78:50156) 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 -- 172.21.15.78:0/3662501158 learned_addr learned my addr 172.21.15.78:0/3662501158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4feb65640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8106a80 0x7ff4f819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fe364640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 0x7ff4f8198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 -- 172.21.15.78:0/3662501158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8106a80 msgr2=0x7ff4f819cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8106a80 0x7ff4f819cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 -- 172.21.15.78:0/3662501158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 msgr2=0x7ff4f8198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 0x7ff4f8198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.025+0000 7ff4fdb63640 1 -- 172.21.15.78:0/3662501158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f819d5c0 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff4fdb63640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800b4a0 tx=0x7ff4e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8014070 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4e8010af0 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f819d8b0 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8017d10 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.026+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f81a5190 con 0x7ff4f8101420 2024-09-16T01:09:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.027+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f8100f60 con 0x7ff4f8101420 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.028+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff4e8017070 con 0x7ff4f8101420 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.028+0000 7ff4ef7fe640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 0x7ff4e40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.028+0000 7ff4fe364640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 0x7ff4e40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.029+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff4e80cf050 con 0x7ff4f8101420 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.029+0000 7ff4fe364640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 0x7ff4e40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40096c0 tx=0x7ff4f4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.032+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4e809bae0 con 0x7ff4f8101420 2024-09-16T01:09:15.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.224+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff4f810a030 con 0x7ff4e40a32d0 2024-09-16T01:09:15.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:15 smithi078 ceph-mon[30937]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:15.227 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:15 smithi078 ceph-mon[30937]: pgmap v307: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:15.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.248+0000 7ff4ef7fe640 1 -- 172.21.15.78:0/3662501158 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff4f810a030 con 0x7ff4e40a32d0 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 msgr2=0x7ff4e40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 0x7ff4e40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40096c0 tx=0x7ff4f4000bc0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 msgr2=0x7ff4f8198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800b4a0 tx=0x7ff4e800b960 comp rx=0 tx=0).stop 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 shutdown_connections 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff4e40a32d0 0x7ff4e40a5790 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4f8106a80 0x7ff4f819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f8101420 0x7ff4f8198b50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 --2- 172.21.15.78:0/3662501158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4f8100a80 0x7ff4f8198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 >> 172.21.15.78:0/3662501158 conn(0x7ff4f80fc7a0 msgr2=0x7ff4f806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 shutdown_connections 2024-09-16T01:09:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:15.251+0000 7ff5005c8640 1 -- 172.21.15.78:0/3662501158 wait complete. 2024-09-16T01:09:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:15 smithi148 ceph-mon[35030]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:15 smithi148 ceph-mon[35030]: pgmap v307: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:15 smithi193 ceph-mon[35701]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:15 smithi193 ceph-mon[35701]: pgmap v307: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:15.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-16T01:09:16.172 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- 172.21.15.78:0/3894649262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102ef0 msgr2=0x7f109410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 --2- 172.21.15.78:0/3894649262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102ef0 0x7f109410da40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f108c009ca0 tx=0x7f108c02a950 comp rx=0 tx=0).stop 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- 172.21.15.78:0/3894649262 shutdown_connections 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 --2- 172.21.15.78:0/3894649262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102ef0 0x7f109410da40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 --2- 172.21.15.78:0/3894649262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102570 0x7f10941029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 --2- 172.21.15.78:0/3894649262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 0x7f1094101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- 172.21.15.78:0/3894649262 >> 172.21.15.78:0/3894649262 conn(0x7f10940fc7d0 msgr2=0x7f10940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- 172.21.15.78:0/3894649262 shutdown_connections 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- 172.21.15.78:0/3894649262 wait complete. 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 Processor -- start 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.669+0000 7f109c18c640 1 -- start start 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 0x7f1094198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 0x7f109419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1094110f80 con 0x7f1094102570 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1094110e00 con 0x7f1094100e20 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1094111100 con 0x7f1094102ef0 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50180/0 (socket says 172.21.15.78:50180) 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 -- 172.21.15.78:0/2176635928 learned_addr learned my addr 172.21.15.78:0/2176635928 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099f28640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 0x7f1094198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109a729640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 0x7f109419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 -- 172.21.15.78:0/2176635928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 msgr2=0x7f109419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 0x7f109419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 -- 172.21.15.78:0/2176635928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 msgr2=0x7f1094198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 0x7f1094198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 -- 172.21.15.78:0/2176635928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109419d4b0 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109a729640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 0x7f109419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f1099727640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f108400d770 tx=0x7f108400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084017070 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109419d7a0 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1084005500 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084004e00 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.670+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10941a5080 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.671+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1094101200 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.672+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1084004fa0 con 0x7f1094102570 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.673+0000 7f108affd640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 0x7f10800a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:16.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.673+0000 7f1099f28640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 0x7f10800a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:16.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.674+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f10840cf810 con 0x7f1094102570 2024-09-16T01:09:16.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.674+0000 7f1099f28640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 0x7f10800a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10900058c0 tx=0x7f109000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:16.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.678+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f108409c130 con 0x7f1094102570 2024-09-16T01:09:16.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.873+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1094062630 con 0x7f10800a32d0 2024-09-16T01:09:16.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.877+0000 7f108affd640 1 -- 172.21.15.78:0/2176635928 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1094062630 con 0x7f10800a32d0 2024-09-16T01:09:16.878 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_1/d9b8d722-5b7a-4d90-86f4-84adec585113 2024-09-16T01:09:16.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 msgr2=0x7f10800a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:16.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 0x7f10800a5790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f10900058c0 tx=0x7f109000a4c0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 msgr2=0x7f1094198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f108400d770 tx=0x7f108400dc30 comp rx=0 tx=0).stop 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 shutdown_connections 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f10800a32d0 0x7f10800a5790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1094102ef0 0x7f109419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1094102570 0x7f1094198a40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 --2- 172.21.15.78:0/2176635928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1094100e20 0x7f1094198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 >> 172.21.15.78:0/2176635928 conn(0x7f10940fc7d0 msgr2=0x7f109406a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 shutdown_connections 2024-09-16T01:09:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:16.879+0000 7f109c18c640 1 -- 172.21.15.78:0/2176635928 wait complete. 2024-09-16T01:09:17.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:17 smithi078 ceph-mon[30937]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:17.197 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:17 smithi078 ceph-mon[30937]: pgmap v308: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:17 smithi148 ceph-mon[35030]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:17 smithi148 ceph-mon[35030]: pgmap v308: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:17 smithi193 ceph-mon[35701]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:17 smithi193 ceph-mon[35701]: pgmap v308: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-09-16T01:09:17.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:09:17.809 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:18.203 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:18 smithi078 ceph-mon[30937]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.337+0000 7f5abbd59640 1 -- 172.21.15.78:0/625323360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4101de0 msgr2=0x7f5ab41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.337+0000 7f5abbd59640 1 --2- 172.21.15.78:0/625323360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4101de0 0x7f5ab41021c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009900 tx=0x7f5ab002a3f0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- 172.21.15.78:0/625323360 shutdown_connections 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- 172.21.15.78:0/625323360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4106f10 0x7f5ab410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- 172.21.15.78:0/625323360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4102790 0x7f5ab41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- 172.21.15.78:0/625323360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4101de0 0x7f5ab41021c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- 172.21.15.78:0/625323360 >> 172.21.15.78:0/625323360 conn(0x7f5ab40fd8c0 msgr2=0x7f5ab40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- 172.21.15.78:0/625323360 shutdown_connections 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- 172.21.15.78:0/625323360 wait complete. 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 Processor -- start 2024-09-16T01:09:18.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- start start 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101de0 0x7f5ab4198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4102790 0x7f5ab4198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4111f70 con 0x7f5ab4106f10 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4111df0 con 0x7f5ab4102790 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.338+0000 7f5abbd59640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41120f0 con 0x7f5ab4101de0 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50202/0 (socket says 172.21.15.78:50202) 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 -- 172.21.15.78:0/709542012 learned_addr learned my addr 172.21.15.78:0/709542012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5ab9af5640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101de0 0x7f5ab4198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5ab92f4640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4102790 0x7f5ab4198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 -- 172.21.15.78:0/709542012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101de0 msgr2=0x7f5ab4198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:18.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101de0 0x7f5ab4198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 -- 172.21.15.78:0/709542012 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4102790 msgr2=0x7f5ab4198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4102790 0x7f5ab4198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 -- 172.21.15.78:0/709542012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab419d2b0 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aba2f6640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00cce0 tx=0x7f5aac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00f450 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aac00fad0 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab419d5a0 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac018b60 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.339+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a4e80 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.340+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab41021c0 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.341+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5aac00fc80 con 0x7f5ab4106f10 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.342+0000 7f5aaaffd640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 0x7f5aa00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:18.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.342+0000 7f5ab9af5640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 0x7f5aa00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:18.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.342+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5aac0d00a0 con 0x7f5ab4106f10 2024-09-16T01:09:18.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.342+0000 7f5ab9af5640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 0x7f5aa00a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:18.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.346+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5aac014030 con 0x7f5ab4106f10 2024-09-16T01:09:18.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:18 smithi148 ceph-mon[35030]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:18.447 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:18 smithi193 ceph-mon[35701]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:09:18.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.548+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab410a150 con 0x7f5ab4106f10 2024-09-16T01:09:18.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.549+0000 7f5aaaffd640 1 -- 172.21.15.78:0/709542012 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7f5aac09ca20 con 0x7f5ab4106f10 2024-09-16T01:09:18.550 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:18.550 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:09:18.550 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:09:18.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 msgr2=0x7f5aa00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:18.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 0x7f5aa00a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab00058d0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 msgr2=0x7f5ab419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:18.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00cce0 tx=0x7f5aac00a620 comp rx=0 tx=0).stop 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 shutdown_connections 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5aa00a3350 0x7f5aa00a5810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ab4106f10 0x7f5ab419cbd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ab4102790 0x7f5ab4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 --2- 172.21.15.78:0/709542012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ab4101de0 0x7f5ab4198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 >> 172.21.15.78:0/709542012 conn(0x7f5ab40fd8c0 msgr2=0x7f5ab40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 shutdown_connections 2024-09-16T01:09:18.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:18.551+0000 7f5abbd59640 1 -- 172.21.15.78:0/709542012 wait complete. 2024-09-16T01:09:19.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:19 smithi078 ceph-mon[30937]: pgmap v309: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 14 op/s 2024-09-16T01:09:19.225 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:19 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/709542012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:09:19.226 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:09:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:19 smithi148 ceph-mon[35030]: pgmap v309: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 14 op/s 2024-09-16T01:09:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:19 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/709542012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:09:19.474 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:19.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:19 smithi193 ceph-mon[35701]: pgmap v309: 129 pgs: 129 active+clean; 609 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 14 op/s 2024-09-16T01:09:19.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:19 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/709542012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:09:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 -- 172.21.15.78:0/268751415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 msgr2=0x7ff5bc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/268751415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc10d950 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac009930 tx=0x7ff5ac02a420 comp rx=0 tx=0).stop 2024-09-16T01:09:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 -- 172.21.15.78:0/268751415 shutdown_connections 2024-09-16T01:09:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/268751415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc10d950 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/268751415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/268751415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.970+0000 7ff5c10d7640 1 -- 172.21.15.78:0/268751415 >> 172.21.15.78:0/268751415 conn(0x7ff5bc0fc7a0 msgr2=0x7ff5bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.971+0000 7ff5c10d7640 1 -- 172.21.15.78:0/268751415 shutdown_connections 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.971+0000 7ff5c10d7640 1 -- 172.21.15.78:0/268751415 wait complete. 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.971+0000 7ff5c10d7640 1 Processor -- start 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 -- start start 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc19bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc110e80 con 0x7ff5bc101420 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc110d00 con 0x7ff5bc100a80 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5c10d7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc111000 con 0x7ff5bc106a80 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50214/0 (socket says 172.21.15.78:50214) 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 -- 172.21.15.78:0/990892113 learned_addr learned my addr 172.21.15.78:0/990892113 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5bb59e640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc19bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5bad9d640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 -- 172.21.15.78:0/990892113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 msgr2=0x7ff5bc19bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc19bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 -- 172.21.15.78:0/990892113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 msgr2=0x7ff5bc197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 -- 172.21.15.78:0/990892113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5bc19c240 con 0x7ff5bc101420 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5bad9d640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5bb59e640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc19bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5ba59c640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c920 tx=0x7ff5a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.972+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4007c70 con 0x7ff5bc101420 2024-09-16T01:09:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.973+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5bc19c530 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.973+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a4007e10 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.973+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4018890 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.973+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc1a3e10 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.974+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5bc100ee0 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.975+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff5a4018a70 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.975+0000 7ff5b3fff640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 0x7ff5a80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.976+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff5a40cfee0 con 0x7ff5bc101420 2024-09-16T01:09:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.976+0000 7ff5bad9d640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 0x7ff5a80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.976+0000 7ff5bad9d640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 0x7ff5a80a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4005fd0 tx=0x7ff5b40093c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:19.979+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff5a4014030 con 0x7ff5bc101420 2024-09-16T01:09:20.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.148+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5bc1a4140 con 0x7ff5bc101420 2024-09-16T01:09:20.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.150+0000 7ff5b3fff640 1 -- 172.21.15.78:0/990892113 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7ff5a400de20 con 0x7ff5bc101420 2024-09-16T01:09:20.151 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:20.151 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 msgr2=0x7ff5a80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 0x7ff5a80a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4005fd0 tx=0x7ff5b40093c0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 msgr2=0x7ff5bc1977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c920 tx=0x7ff5a400cde0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 shutdown_connections 2024-09-16T01:09:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff5a80a32d0 0x7ff5a80a5790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5bc106a80 0x7ff5bc19bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5bc101420 0x7ff5bc1977d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 --2- 172.21.15.78:0/990892113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5bc100a80 0x7ff5bc197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 >> 172.21.15.78:0/990892113 conn(0x7ff5bc0fc7a0 msgr2=0x7ff5bc0fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 shutdown_connections 2024-09-16T01:09:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:20.152+0000 7ff5c10d7640 1 -- 172.21.15.78:0/990892113 wait complete. 2024-09-16T01:09:20.867 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T01:09:20.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:09:21.113 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:21.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:21 smithi078 ceph-mon[30937]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-09-16T01:09:21.234 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:21 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/990892113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:21 smithi148 ceph-mon[35030]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-09-16T01:09:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:21 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/990892113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:21.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:21 smithi193 ceph-mon[35701]: pgmap v310: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-09-16T01:09:21.557 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:21 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/990892113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 -- 172.21.15.78:0/447509763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 msgr2=0x7f993c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 --2- 172.21.15.78:0/447509763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c10ea90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9934009ca0 tx=0x7f993402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 -- 172.21.15.78:0/447509763 shutdown_connections 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 --2- 172.21.15.78:0/447509763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c10ea90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 --2- 172.21.15.78:0/447509763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 --2- 172.21.15.78:0/447509763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 -- 172.21.15.78:0/447509763 >> 172.21.15.78:0/447509763 conn(0x7f993c0fd8c0 msgr2=0x7f993c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 -- 172.21.15.78:0/447509763 shutdown_connections 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.597+0000 7f9942224640 1 -- 172.21.15.78:0/447509763 wait complete. 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 Processor -- start 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 -- start start 2024-09-16T01:09:21.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f993c111f70 con 0x7f993c106f10 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f993c111df0 con 0x7f993c102790 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f9942224640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1120f0 con 0x7f993c101de0 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f993bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f993bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50240/0 (socket says 172.21.15.78:50240) 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f993bfff640 1 -- 172.21.15.78:0/1839513152 learned_addr learned my addr 172.21.15.78:0/1839513152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.598+0000 7f993affd640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993b7fe640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:21.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 -- 172.21.15.78:0/1839513152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 msgr2=0x7f993c198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 -- 172.21.15.78:0/1839513152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 msgr2=0x7f993c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 -- 172.21.15.78:0/1839513152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c19d300 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993b7fe640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993affd640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f993bfff640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9934031a60 tx=0x7f9934031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934003c30 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c19d590 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9934003dd0 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99340047e0 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.599+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c1a4ed0 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.600+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f993c1021c0 con 0x7f993c106f10 2024-09-16T01:09:21.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.601+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f993402f790 con 0x7f993c106f10 2024-09-16T01:09:21.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.602+0000 7f9938ff9640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 0x7f99240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:21.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.602+0000 7f993b7fe640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 0x7f99240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:21.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.602+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f99340f10b0 con 0x7f993c106f10 2024-09-16T01:09:21.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.602+0000 7f993b7fe640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 0x7f99240a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f992c0058c0 tx=0x7f992c005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:21.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.606+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f99340bdb40 con 0x7f993c106f10 2024-09-16T01:09:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.771+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f993c0fec00 con 0x7f99240a32d0 2024-09-16T01:09:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.780+0000 7f9938ff9640 1 -- 172.21.15.78:0/1839513152 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340611 (secure 0 0 0) 0x7f993c0fec00 con 0x7f99240a32d0 2024-09-16T01:09:21.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:21.797 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:09:21.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 msgr2=0x7f99240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:21.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 0x7f99240a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f992c0058c0 tx=0x7f992c005d30 comp rx=0 tx=0).stop 2024-09-16T01:09:21.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 msgr2=0x7f993c19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:21.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9934031a60 tx=0x7f9934031a90 comp rx=0 tx=0).stop 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 shutdown_connections 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f99240a32d0 0x7f99240a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f993c106f10 0x7f993c19cc20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f993c102790 0x7f993c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 --2- 172.21.15.78:0/1839513152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f993c101de0 0x7f993c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:21.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 >> 172.21.15.78:0/1839513152 conn(0x7f993c0fd8c0 msgr2=0x7f993c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:21.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.783+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 shutdown_connections 2024-09-16T01:09:21.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:21.784+0000 7f9942224640 1 -- 172.21.15.78:0/1839513152 wait complete. 2024-09-16T01:09:22.485 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T01:09:20.003697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647013,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":230,"num_read_kb":219,"num_write":342,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373588,"kb_used_data":11860,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568236,"statfs":{"total":1151940427776,"available":1151557873664,"internally_reserved":0,"allocated":12144640,"data_stored":7761184,"data_compressed":318360,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49281,"num_objects":11,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":87,"num_read_kb":94,"num_write":53,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103415+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:09:05.103415+0000","last_peered":"2024-09-16T01:09:05.103415+0000","last_clean":"2024-09-16T01:09:05.103415+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:09:05.103415+0000","last_undegraded":"2024-09-16T01:09:05.103415+0000","last_fullsized":"2024-09-16T01:09:05.103415+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235039+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:08:39.235039+0000","last_peered":"2024-09-16T01:08:39.235039+0000","last_clean":"2024-09-16T01:08:39.235039+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:08:39.235039+0000","last_undegraded":"2024-09-16T01:08:39.235039+0000","last_fullsized":"2024-09-16T01:08:39.235039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769984+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:08:38.769984+0000","last_peered":"2024-09-16T01:08:38.769984+0000","last_clean":"2024-09-16T01:08:38.769984+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:08:38.769984+0000","last_undegraded":"2024-09-16T01:08:38.769984+0000","last_fullsized":"2024-09-16T01:08:38.769984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235320+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:08:39.235320+0000","last_peered":"2024-09-16T01:08:39.235320+0000","last_clean":"2024-09-16T01:08:39.235320+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:08:39.235320+0000","last_undegraded":"2024-09-16T01:08:39.235320+0000","last_fullsized":"2024-09-16T01:08:39.235320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236157+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:08:39.236157+0000","last_peered":"2024-09-16T01:08:39.236157+0000","last_clean":"2024-09-16T01:08:39.236157+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:08:39.236157+0000","last_undegraded":"2024-09-16T01:08:39.236157+0000","last_fullsized":"2024-09-16T01:08:39.236157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235231+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:08:39.235231+0000","last_peered":"2024-09-16T01:08:39.235231+0000","last_clean":"2024-09-16T01:08:39.235231+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:08:39.235231+0000","last_undegraded":"2024-09-16T01:08:39.235231+0000","last_fullsized":"2024-09-16T01:08:39.235231+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102254+0000","last_change":"2024-09-16T01:08:26.711638+0000","last_active":"2024-09-16T01:09:05.102254+0000","last_peered":"2024-09-16T01:09:05.102254+0000","last_clean":"2024-09-16T01:09:05.102254+0000","last_became_active":"2024-09-16T01:08:26.711038+0000","last_became_peered":"2024-09-16T01:08:26.711038+0000","last_unstale":"2024-09-16T01:09:05.102254+0000","last_undegraded":"2024-09-16T01:09:05.102254+0000","last_fullsized":"2024-09-16T01:09:05.102254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:12.619264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103270+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:08:39.103270+0000","last_peered":"2024-09-16T01:08:39.103270+0000","last_clean":"2024-09-16T01:08:39.103270+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:08:39.103270+0000","last_undegraded":"2024-09-16T01:08:39.103270+0000","last_fullsized":"2024-09-16T01:08:39.103270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236249+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:08:39.236249+0000","last_peered":"2024-09-16T01:08:39.236249+0000","last_clean":"2024-09-16T01:08:39.236249+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:08:39.236249+0000","last_undegraded":"2024-09-16T01:08:39.236249+0000","last_fullsized":"2024-09-16T01:08:39.236249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769910+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:08:38.769910+0000","last_peered":"2024-09-16T01:08:38.769910+0000","last_clean":"2024-09-16T01:08:38.769910+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:08:38.769910+0000","last_undegraded":"2024-09-16T01:08:38.769910+0000","last_fullsized":"2024-09-16T01:08:38.769910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236351+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:08:39.236351+0000","last_peered":"2024-09-16T01:08:39.236351+0000","last_clean":"2024-09-16T01:08:39.236351+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:08:39.236351+0000","last_undegraded":"2024-09-16T01:08:39.236351+0000","last_fullsized":"2024-09-16T01:08:39.236351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235876+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:08:39.235876+0000","last_peered":"2024-09-16T01:08:39.235876+0000","last_clean":"2024-09-16T01:08:39.235876+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:08:39.235876+0000","last_undegraded":"2024-09-16T01:08:39.235876+0000","last_fullsized":"2024-09-16T01:08:39.235876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235379+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:08:39.235379+0000","last_peered":"2024-09-16T01:08:39.235379+0000","last_clean":"2024-09-16T01:08:39.235379+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:08:39.235379+0000","last_undegraded":"2024-09-16T01:08:39.235379+0000","last_fullsized":"2024-09-16T01:08:39.235379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.095373+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:09:04.095373+0000","last_peered":"2024-09-16T01:09:04.095373+0000","last_clean":"2024-09-16T01:09:04.095373+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:09:04.095373+0000","last_undegraded":"2024-09-16T01:09:04.095373+0000","last_fullsized":"2024-09-16T01:09:04.095373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236158+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:08:39.236158+0000","last_peered":"2024-09-16T01:08:39.236158+0000","last_clean":"2024-09-16T01:08:39.236158+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:08:39.236158+0000","last_undegraded":"2024-09-16T01:08:39.236158+0000","last_fullsized":"2024-09-16T01:08:39.236158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235421+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:08:39.235421+0000","last_peered":"2024-09-16T01:08:39.235421+0000","last_clean":"2024-09-16T01:08:39.235421+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:08:39.235421+0000","last_undegraded":"2024-09-16T01:08:39.235421+0000","last_fullsized":"2024-09-16T01:08:39.235421+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770368+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:08:38.770368+0000","last_peered":"2024-09-16T01:08:38.770368+0000","last_clean":"2024-09-16T01:08:38.770368+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:08:38.770368+0000","last_undegraded":"2024-09-16T01:08:38.770368+0000","last_fullsized":"2024-09-16T01:08:38.770368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236429+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:08:39.236429+0000","last_peered":"2024-09-16T01:08:39.236429+0000","last_clean":"2024-09-16T01:08:39.236429+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:08:39.236429+0000","last_undegraded":"2024-09-16T01:08:39.236429+0000","last_fullsized":"2024-09-16T01:08:39.236429+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235319+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:08:39.235319+0000","last_peered":"2024-09-16T01:08:39.235319+0000","last_clean":"2024-09-16T01:08:39.235319+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:08:39.235319+0000","last_undegraded":"2024-09-16T01:08:39.235319+0000","last_fullsized":"2024-09-16T01:08:39.235319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094233+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:09:04.094233+0000","last_peered":"2024-09-16T01:09:04.094233+0000","last_clean":"2024-09-16T01:09:04.094233+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:09:04.094233+0000","last_undegraded":"2024-09-16T01:09:04.094233+0000","last_fullsized":"2024-09-16T01:09:04.094233+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236710+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:08:39.236710+0000","last_peered":"2024-09-16T01:08:39.236710+0000","last_clean":"2024-09-16T01:08:39.236710+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:08:39.236710+0000","last_undegraded":"2024-09-16T01:08:39.236710+0000","last_fullsized":"2024-09-16T01:08:39.236710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769023+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:08:38.769023+0000","last_peered":"2024-09-16T01:08:38.769023+0000","last_clean":"2024-09-16T01:08:38.769023+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:08:38.769023+0000","last_undegraded":"2024-09-16T01:08:38.769023+0000","last_fullsized":"2024-09-16T01:08:38.769023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094629+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:09:04.094629+0000","last_peered":"2024-09-16T01:09:04.094629+0000","last_clean":"2024-09-16T01:09:04.094629+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:09:04.094629+0000","last_undegraded":"2024-09-16T01:09:04.094629+0000","last_fullsized":"2024-09-16T01:09:04.094629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235926+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:08:39.235926+0000","last_peered":"2024-09-16T01:08:39.235926+0000","last_clean":"2024-09-16T01:08:39.235926+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:08:39.235926+0000","last_undegraded":"2024-09-16T01:08:39.235926+0000","last_fullsized":"2024-09-16T01:08:39.235926+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235938+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:08:39.235938+0000","last_peered":"2024-09-16T01:08:39.235938+0000","last_clean":"2024-09-16T01:08:39.235938+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:08:39.235938+0000","last_undegraded":"2024-09-16T01:08:39.235938+0000","last_fullsized":"2024-09-16T01:08:39.235938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093144+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:09:04.093144+0000","last_peered":"2024-09-16T01:09:04.093144+0000","last_clean":"2024-09-16T01:09:04.093144+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:09:04.093144+0000","last_undegraded":"2024-09-16T01:09:04.093144+0000","last_fullsized":"2024-09-16T01:09:04.093144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770220+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:08:38.770220+0000","last_peered":"2024-09-16T01:08:38.770220+0000","last_clean":"2024-09-16T01:08:38.770220+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:08:38.770220+0000","last_undegraded":"2024-09-16T01:08:38.770220+0000","last_fullsized":"2024-09-16T01:08:38.770220+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235280+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:08:39.235280+0000","last_peered":"2024-09-16T01:08:39.235280+0000","last_clean":"2024-09-16T01:08:39.235280+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:08:39.235280+0000","last_undegraded":"2024-09-16T01:08:39.235280+0000","last_fullsized":"2024-09-16T01:08:39.235280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:15.743094+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:09:15.743094+0000","last_peered":"2024-09-16T01:09:15.743094+0000","last_clean":"2024-09-16T01:09:15.743094+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:09:15.743094+0000","last_undegraded":"2024-09-16T01:09:15.743094+0000","last_fullsized":"2024-09-16T01:09:15.743094+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.234938+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:08:39.234938+0000","last_peered":"2024-09-16T01:08:39.234938+0000","last_clean":"2024-09-16T01:08:39.234938+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:08:39.234938+0000","last_undegraded":"2024-09-16T01:08:39.234938+0000","last_fullsized":"2024-09-16T01:08:39.234938+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236601+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:08:39.236601+0000","last_peered":"2024-09-16T01:08:39.236601+0000","last_clean":"2024-09-16T01:08:39.236601+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:08:39.236601+0000","last_undegraded":"2024-09-16T01:08:39.236601+0000","last_fullsized":"2024-09-16T01:08:39.236601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093321+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:09:04.093321+0000","last_peered":"2024-09-16T01:09:04.093321+0000","last_clean":"2024-09-16T01:09:04.093321+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:09:04.093321+0000","last_undegraded":"2024-09-16T01:09:04.093321+0000","last_fullsized":"2024-09-16T01:09:04.093321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235229+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:08:39.235229+0000","last_peered":"2024-09-16T01:08:39.235229+0000","last_clean":"2024-09-16T01:08:39.235229+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:08:39.235229+0000","last_undegraded":"2024-09-16T01:08:39.235229+0000","last_fullsized":"2024-09-16T01:08:39.235229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772709+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:08:38.772709+0000","last_peered":"2024-09-16T01:08:38.772709+0000","last_clean":"2024-09-16T01:08:38.772709+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:08:38.772709+0000","last_undegraded":"2024-09-16T01:08:38.772709+0000","last_fullsized":"2024-09-16T01:08:38.772709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236879+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:08:39.236879+0000","last_peered":"2024-09-16T01:08:39.236879+0000","last_clean":"2024-09-16T01:08:39.236879+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:08:39.236879+0000","last_undegraded":"2024-09-16T01:08:39.236879+0000","last_fullsized":"2024-09-16T01:08:39.236879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769206+0000","last_change":"2024-09-16T01:08:36.345546+0000","last_active":"2024-09-16T01:08:38.769206+0000","last_peered":"2024-09-16T01:08:38.769206+0000","last_clean":"2024-09-16T01:08:38.769206+0000","last_became_active":"2024-09-16T01:08:36.344943+0000","last_became_peered":"2024-09-16T01:08:36.344943+0000","last_unstale":"2024-09-16T01:08:38.769206+0000","last_undegraded":"2024-09-16T01:08:38.769206+0000","last_fullsized":"2024-09-16T01:08:38.769206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:32.997745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772775+0000","last_change":"2024-09-16T01:08:28.715546+0000","last_active":"2024-09-16T01:08:38.772775+0000","last_peered":"2024-09-16T01:08:38.772775+0000","last_clean":"2024-09-16T01:08:38.772775+0000","last_became_active":"2024-09-16T01:08:28.714380+0000","last_became_peered":"2024-09-16T01:08:28.714380+0000","last_unstale":"2024-09-16T01:08:38.772775+0000","last_undegraded":"2024-09-16T01:08:38.772775+0000","last_fullsized":"2024-09-16T01:08:38.772775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:05.884119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235658+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:08:39.235658+0000","last_peered":"2024-09-16T01:08:39.235658+0000","last_clean":"2024-09-16T01:08:39.235658+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:08:39.235658+0000","last_undegraded":"2024-09-16T01:08:39.235658+0000","last_fullsized":"2024-09-16T01:08:39.235658+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235479+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:08:39.235479+0000","last_peered":"2024-09-16T01:08:39.235479+0000","last_clean":"2024-09-16T01:08:39.235479+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:08:39.235479+0000","last_undegraded":"2024-09-16T01:08:39.235479+0000","last_fullsized":"2024-09-16T01:08:39.235479+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235107+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:08:39.235107+0000","last_peered":"2024-09-16T01:08:39.235107+0000","last_clean":"2024-09-16T01:08:39.235107+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:08:39.235107+0000","last_undegraded":"2024-09-16T01:08:39.235107+0000","last_fullsized":"2024-09-16T01:08:39.235107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:16.131359+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:09:16.131359+0000","last_peered":"2024-09-16T01:09:16.131359+0000","last_clean":"2024-09-16T01:09:16.131359+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:09:16.131359+0000","last_undegraded":"2024-09-16T01:09:16.131359+0000","last_fullsized":"2024-09-16T01:09:16.131359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236906+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:08:39.236906+0000","last_peered":"2024-09-16T01:08:39.236906+0000","last_clean":"2024-09-16T01:08:39.236906+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:08:39.236906+0000","last_undegraded":"2024-09-16T01:08:39.236906+0000","last_fullsized":"2024-09-16T01:08:39.236906+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235482+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:08:39.235482+0000","last_peered":"2024-09-16T01:08:39.235482+0000","last_clean":"2024-09-16T01:08:39.235482+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:08:39.235482+0000","last_undegraded":"2024-09-16T01:08:39.235482+0000","last_fullsized":"2024-09-16T01:08:39.235482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102908+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:09:05.102908+0000","last_peered":"2024-09-16T01:09:05.102908+0000","last_clean":"2024-09-16T01:09:05.102908+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:09:05.102908+0000","last_undegraded":"2024-09-16T01:09:05.102908+0000","last_fullsized":"2024-09-16T01:09:05.102908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235641+0000","last_change":"2024-09-16T01:08:36.346613+0000","last_active":"2024-09-16T01:08:39.235641+0000","last_peered":"2024-09-16T01:08:39.235641+0000","last_clean":"2024-09-16T01:08:39.235641+0000","last_became_active":"2024-09-16T01:08:36.346205+0000","last_became_peered":"2024-09-16T01:08:36.346205+0000","last_unstale":"2024-09-16T01:08:39.235641+0000","last_undegraded":"2024-09-16T01:08:39.235641+0000","last_fullsized":"2024-09-16T01:08:39.235641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:36.409657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235748+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:08:39.235748+0000","last_peered":"2024-09-16T01:08:39.235748+0000","last_clean":"2024-09-16T01:08:39.235748+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:08:39.235748+0000","last_undegraded":"2024-09-16T01:08:39.235748+0000","last_fullsized":"2024-09-16T01:08:39.235748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:06.111839+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:09:06.111839+0000","last_peered":"2024-09-16T01:09:06.111839+0000","last_clean":"2024-09-16T01:09:06.111839+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:09:06.111839+0000","last_undegraded":"2024-09-16T01:09:06.111839+0000","last_fullsized":"2024-09-16T01:09:06.111839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235851+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:08:39.235851+0000","last_peered":"2024-09-16T01:08:39.235851+0000","last_clean":"2024-09-16T01:08:39.235851+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:08:39.235851+0000","last_undegraded":"2024-09-16T01:08:39.235851+0000","last_fullsized":"2024-09-16T01:08:39.235851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236098+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:08:39.236098+0000","last_peered":"2024-09-16T01:08:39.236098+0000","last_clean":"2024-09-16T01:08:39.236098+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:08:39.236098+0000","last_undegraded":"2024-09-16T01:08:39.236098+0000","last_fullsized":"2024-09-16T01:08:39.236098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235848+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:08:39.235848+0000","last_peered":"2024-09-16T01:08:39.235848+0000","last_clean":"2024-09-16T01:08:39.235848+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:08:39.235848+0000","last_undegraded":"2024-09-16T01:08:39.235848+0000","last_fullsized":"2024-09-16T01:08:39.235848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103616+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:09:05.103616+0000","last_peered":"2024-09-16T01:09:05.103616+0000","last_clean":"2024-09-16T01:09:05.103616+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:09:05.103616+0000","last_undegraded":"2024-09-16T01:09:05.103616+0000","last_fullsized":"2024-09-16T01:09:05.103616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103160+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:08:39.103160+0000","last_peered":"2024-09-16T01:08:39.103160+0000","last_clean":"2024-09-16T01:08:39.103160+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:08:39.103160+0000","last_undegraded":"2024-09-16T01:08:39.103160+0000","last_fullsized":"2024-09-16T01:08:39.103160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235590+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:08:39.235590+0000","last_peered":"2024-09-16T01:08:39.235590+0000","last_clean":"2024-09-16T01:08:39.235590+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:08:39.235590+0000","last_undegraded":"2024-09-16T01:08:39.235590+0000","last_fullsized":"2024-09-16T01:08:39.235590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236582+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:08:39.236582+0000","last_peered":"2024-09-16T01:08:39.236582+0000","last_clean":"2024-09-16T01:08:39.236582+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:08:39.236582+0000","last_undegraded":"2024-09-16T01:08:39.236582+0000","last_fullsized":"2024-09-16T01:08:39.236582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235554+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:08:39.235554+0000","last_peered":"2024-09-16T01:08:39.235554+0000","last_clean":"2024-09-16T01:08:39.235554+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:08:39.235554+0000","last_undegraded":"2024-09-16T01:08:39.235554+0000","last_fullsized":"2024-09-16T01:08:39.235554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235615+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:08:39.235615+0000","last_peered":"2024-09-16T01:08:39.235615+0000","last_clean":"2024-09-16T01:08:39.235615+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:08:39.235615+0000","last_undegraded":"2024-09-16T01:08:39.235615+0000","last_fullsized":"2024-09-16T01:08:39.235615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.237232+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:08:39.237232+0000","last_peered":"2024-09-16T01:08:39.237232+0000","last_clean":"2024-09-16T01:08:39.237232+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:08:39.237232+0000","last_undegraded":"2024-09-16T01:08:39.237232+0000","last_fullsized":"2024-09-16T01:08:39.237232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235027+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:08:39.235027+0000","last_peered":"2024-09-16T01:08:39.235027+0000","last_clean":"2024-09-16T01:08:39.235027+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:08:39.235027+0000","last_undegraded":"2024-09-16T01:08:39.235027+0000","last_fullsized":"2024-09-16T01:08:39.235027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235324+0000","last_change":"2024-09-16T01:08:28.726332+0000","last_active":"2024-09-16T01:08:39.235324+0000","last_peered":"2024-09-16T01:08:39.235324+0000","last_clean":"2024-09-16T01:08:39.235324+0000","last_became_active":"2024-09-16T01:08:28.725901+0000","last_became_peered":"2024-09-16T01:08:28.725901+0000","last_unstale":"2024-09-16T01:08:39.235324+0000","last_undegraded":"2024-09-16T01:08:39.235324+0000","last_fullsized":"2024-09-16T01:08:39.235324+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:18:57.922805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:06.111851+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:09:06.111851+0000","last_peered":"2024-09-16T01:09:06.111851+0000","last_clean":"2024-09-16T01:09:06.111851+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:09:06.111851+0000","last_undegraded":"2024-09-16T01:09:06.111851+0000","last_fullsized":"2024-09-16T01:09:06.111851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235033+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:08:39.235033+0000","last_peered":"2024-09-16T01:08:39.235033+0000","last_clean":"2024-09-16T01:08:39.235033+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:08:39.235033+0000","last_undegraded":"2024-09-16T01:08:39.235033+0000","last_fullsized":"2024-09-16T01:08:39.235033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235295+0000","last_change":"2024-09-16T01:08:28.728462+0000","last_active":"2024-09-16T01:08:39.235295+0000","last_peered":"2024-09-16T01:08:39.235295+0000","last_clean":"2024-09-16T01:08:39.235295+0000","last_became_active":"2024-09-16T01:08:28.726704+0000","last_became_peered":"2024-09-16T01:08:28.726704+0000","last_unstale":"2024-09-16T01:08:39.235295+0000","last_undegraded":"2024-09-16T01:08:39.235295+0000","last_fullsized":"2024-09-16T01:08:39.235295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:33:27.278382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103589+0000","last_change":"2024-09-16T01:08:26.956123+0000","last_active":"2024-09-16T01:09:05.103589+0000","last_peered":"2024-09-16T01:09:05.103589+0000","last_clean":"2024-09-16T01:09:05.103589+0000","last_became_active":"2024-09-16T01:08:26.955746+0000","last_became_peered":"2024-09-16T01:08:26.955746+0000","last_unstale":"2024-09-16T01:09:05.103589+0000","last_undegraded":"2024-09-16T01:09:05.103589+0000","last_fullsized":"2024-09-16T01:09:05.103589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:40.575736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.237265+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:08:39.237265+0000","last_peered":"2024-09-16T01:08:39.237265+0000","last_clean":"2024-09-16T01:08:39.237265+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:08:39.237265+0000","last_undegraded":"2024-09-16T01:08:39.237265+0000","last_fullsized":"2024-09-16T01:08:39.237265+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235392+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:08:39.235392+0000","last_peered":"2024-09-16T01:08:39.235392+0000","last_clean":"2024-09-16T01:08:39.235392+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:08:39.235392+0000","last_undegraded":"2024-09-16T01:08:39.235392+0000","last_fullsized":"2024-09-16T01:08:39.235392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:14.129176+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:09:14.129176+0000","last_peered":"2024-09-16T01:09:14.129176+0000","last_clean":"2024-09-16T01:09:14.129176+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:09:14.129176+0000","last_undegraded":"2024-09-16T01:09:14.129176+0000","last_fullsized":"2024-09-16T01:09:14.129176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236957+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:08:39.236957+0000","last_peered":"2024-09-16T01:08:39.236957+0000","last_clean":"2024-09-16T01:08:39.236957+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:08:39.236957+0000","last_undegraded":"2024-09-16T01:08:39.236957+0000","last_fullsized":"2024-09-16T01:08:39.236957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235492+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:08:39.235492+0000","last_peered":"2024-09-16T01:08:39.235492+0000","last_clean":"2024-09-16T01:08:39.235492+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:08:39.235492+0000","last_undegraded":"2024-09-16T01:08:39.235492+0000","last_fullsized":"2024-09-16T01:08:39.235492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.095874+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:09:04.095874+0000","last_peered":"2024-09-16T01:09:04.095874+0000","last_clean":"2024-09-16T01:09:04.095874+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:09:04.095874+0000","last_undegraded":"2024-09-16T01:09:04.095874+0000","last_fullsized":"2024-09-16T01:09:04.095874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235390+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:08:39.235390+0000","last_peered":"2024-09-16T01:08:39.235390+0000","last_clean":"2024-09-16T01:08:39.235390+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:08:39.235390+0000","last_undegraded":"2024-09-16T01:08:39.235390+0000","last_fullsized":"2024-09-16T01:08:39.235390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235981+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:08:39.235981+0000","last_peered":"2024-09-16T01:08:39.235981+0000","last_clean":"2024-09-16T01:08:39.235981+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:08:39.235981+0000","last_undegraded":"2024-09-16T01:08:39.235981+0000","last_fullsized":"2024-09-16T01:08:39.235981+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102595+0000","last_change":"2024-09-16T01:08:26.710927+0000","last_active":"2024-09-16T01:09:05.102595+0000","last_peered":"2024-09-16T01:09:05.102595+0000","last_clean":"2024-09-16T01:09:05.102595+0000","last_became_active":"2024-09-16T01:08:26.709796+0000","last_became_peered":"2024-09-16T01:08:26.709796+0000","last_unstale":"2024-09-16T01:09:05.102595+0000","last_undegraded":"2024-09-16T01:09:05.102595+0000","last_fullsized":"2024-09-16T01:09:05.102595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:30.270120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236238+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:08:39.236238+0000","last_peered":"2024-09-16T01:08:39.236238+0000","last_clean":"2024-09-16T01:08:39.236238+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:08:39.236238+0000","last_undegraded":"2024-09-16T01:08:39.236238+0000","last_fullsized":"2024-09-16T01:08:39.236238+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769443+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:08:38.769443+0000","last_peered":"2024-09-16T01:08:38.769443+0000","last_clean":"2024-09-16T01:08:38.769443+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:08:38.769443+0000","last_undegraded":"2024-09-16T01:08:38.769443+0000","last_fullsized":"2024-09-16T01:08:38.769443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:17.779582+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:09:17.779582+0000","last_peered":"2024-09-16T01:09:17.779582+0000","last_clean":"2024-09-16T01:09:17.779582+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:09:17.779582+0000","last_undegraded":"2024-09-16T01:09:17.779582+0000","last_fullsized":"2024-09-16T01:09:17.779582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52408,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":57,"num_write":30,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236446+0000","last_change":"2024-09-16T01:08:35.768204+0000","last_active":"2024-09-16T01:08:39.236446+0000","last_peered":"2024-09-16T01:08:39.236446+0000","last_clean":"2024-09-16T01:08:39.236446+0000","last_became_active":"2024-09-16T01:08:35.767845+0000","last_became_peered":"2024-09-16T01:08:35.767845+0000","last_unstale":"2024-09-16T01:08:39.236446+0000","last_undegraded":"2024-09-16T01:08:39.236446+0000","last_fullsized":"2024-09-16T01:08:39.236446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:20.646352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103420+0000","last_change":"2024-09-16T01:08:28.725923+0000","last_active":"2024-09-16T01:08:39.103420+0000","last_peered":"2024-09-16T01:08:39.103420+0000","last_clean":"2024-09-16T01:08:39.103420+0000","last_became_active":"2024-09-16T01:08:28.725595+0000","last_became_peered":"2024-09-16T01:08:28.725595+0000","last_unstale":"2024-09-16T01:08:39.103420+0000","last_undegraded":"2024-09-16T01:08:39.103420+0000","last_fullsized":"2024-09-16T01:08:39.103420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:22.000211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093415+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:09:04.093415+0000","last_peered":"2024-09-16T01:09:04.093415+0000","last_clean":"2024-09-16T01:09:04.093415+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:09:04.093415+0000","last_undegraded":"2024-09-16T01:09:04.093415+0000","last_fullsized":"2024-09-16T01:09:04.093415+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236524+0000","last_change":"2024-09-16T01:08:35.768360+0000","last_active":"2024-09-16T01:08:39.236524+0000","last_peered":"2024-09-16T01:08:39.236524+0000","last_clean":"2024-09-16T01:08:39.236524+0000","last_became_active":"2024-09-16T01:08:35.767916+0000","last_became_peered":"2024-09-16T01:08:35.767916+0000","last_unstale":"2024-09-16T01:08:39.236524+0000","last_undegraded":"2024-09-16T01:08:39.236524+0000","last_fullsized":"2024-09-16T01:08:39.236524+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:57.329387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236122+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:08:39.236122+0000","last_peered":"2024-09-16T01:08:39.236122+0000","last_clean":"2024-09-16T01:08:39.236122+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:08:39.236122+0000","last_undegraded":"2024-09-16T01:08:39.236122+0000","last_fullsized":"2024-09-16T01:08:39.236122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094940+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:09:04.094940+0000","last_peered":"2024-09-16T01:09:04.094940+0000","last_clean":"2024-09-16T01:09:04.094940+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:09:04.094940+0000","last_undegraded":"2024-09-16T01:09:04.094940+0000","last_fullsized":"2024-09-16T01:09:04.094940+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236917+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:08:39.236917+0000","last_peered":"2024-09-16T01:08:39.236917+0000","last_clean":"2024-09-16T01:08:39.236917+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:08:39.236917+0000","last_undegraded":"2024-09-16T01:08:39.236917+0000","last_fullsized":"2024-09-16T01:08:39.236917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103442+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:08:39.103442+0000","last_peered":"2024-09-16T01:08:39.103442+0000","last_clean":"2024-09-16T01:08:39.103442+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:08:39.103442+0000","last_undegraded":"2024-09-16T01:08:39.103442+0000","last_fullsized":"2024-09-16T01:08:39.103442+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:16.131934+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:09:16.131934+0000","last_peered":"2024-09-16T01:09:16.131934+0000","last_clean":"2024-09-16T01:09:16.131934+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:09:16.131934+0000","last_undegraded":"2024-09-16T01:09:16.131934+0000","last_fullsized":"2024-09-16T01:09:16.131934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235618+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:08:39.235618+0000","last_peered":"2024-09-16T01:08:39.235618+0000","last_clean":"2024-09-16T01:08:39.235618+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:08:39.235618+0000","last_undegraded":"2024-09-16T01:08:39.235618+0000","last_fullsized":"2024-09-16T01:08:39.235618+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235903+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:08:39.235903+0000","last_peered":"2024-09-16T01:08:39.235903+0000","last_clean":"2024-09-16T01:08:39.235903+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:08:39.235903+0000","last_undegraded":"2024-09-16T01:08:39.235903+0000","last_fullsized":"2024-09-16T01:08:39.235903+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236428+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:08:39.236428+0000","last_peered":"2024-09-16T01:08:39.236428+0000","last_clean":"2024-09-16T01:08:39.236428+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:08:39.236428+0000","last_undegraded":"2024-09-16T01:08:39.236428+0000","last_fullsized":"2024-09-16T01:08:39.236428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235323+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:08:39.235323+0000","last_peered":"2024-09-16T01:08:39.235323+0000","last_clean":"2024-09-16T01:08:39.235323+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:08:39.235323+0000","last_undegraded":"2024-09-16T01:08:39.235323+0000","last_fullsized":"2024-09-16T01:08:39.235323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235729+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:08:39.235729+0000","last_peered":"2024-09-16T01:08:39.235729+0000","last_clean":"2024-09-16T01:08:39.235729+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:08:39.235729+0000","last_undegraded":"2024-09-16T01:08:39.235729+0000","last_fullsized":"2024-09-16T01:08:39.235729+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235195+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:08:39.235195+0000","last_peered":"2024-09-16T01:08:39.235195+0000","last_clean":"2024-09-16T01:08:39.235195+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:08:39.235195+0000","last_undegraded":"2024-09-16T01:08:39.235195+0000","last_fullsized":"2024-09-16T01:08:39.235195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102231+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:09:05.102231+0000","last_peered":"2024-09-16T01:09:05.102231+0000","last_clean":"2024-09-16T01:09:05.102231+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:09:05.102231+0000","last_undegraded":"2024-09-16T01:09:05.102231+0000","last_fullsized":"2024-09-16T01:09:05.102231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102278+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:09:05.102278+0000","last_peered":"2024-09-16T01:09:05.102278+0000","last_clean":"2024-09-16T01:09:05.102278+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:09:05.102278+0000","last_undegraded":"2024-09-16T01:09:05.102278+0000","last_fullsized":"2024-09-16T01:09:05.102278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235768+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:08:39.235768+0000","last_peered":"2024-09-16T01:08:39.235768+0000","last_clean":"2024-09-16T01:08:39.235768+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:08:39.235768+0000","last_undegraded":"2024-09-16T01:08:39.235768+0000","last_fullsized":"2024-09-16T01:08:39.235768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770560+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:08:38.770560+0000","last_peered":"2024-09-16T01:08:38.770560+0000","last_clean":"2024-09-16T01:08:38.770560+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:08:38.770560+0000","last_undegraded":"2024-09-16T01:08:38.770560+0000","last_fullsized":"2024-09-16T01:08:38.770560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:15.131166+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:09:15.131166+0000","last_peered":"2024-09-16T01:09:15.131166+0000","last_clean":"2024-09-16T01:09:15.131166+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:09:15.131166+0000","last_undegraded":"2024-09-16T01:09:15.131166+0000","last_fullsized":"2024-09-16T01:09:15.131166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769178+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:08:38.769178+0000","last_peered":"2024-09-16T01:08:38.769178+0000","last_clean":"2024-09-16T01:08:38.769178+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:08:38.769178+0000","last_undegraded":"2024-09-16T01:08:38.769178+0000","last_fullsized":"2024-09-16T01:08:38.769178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770664+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:08:38.770664+0000","last_peered":"2024-09-16T01:08:38.770664+0000","last_clean":"2024-09-16T01:08:38.770664+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:08:38.770664+0000","last_undegraded":"2024-09-16T01:08:38.770664+0000","last_fullsized":"2024-09-16T01:08:38.770664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235095+0000","last_change":"2024-09-16T01:08:28.719916+0000","last_active":"2024-09-16T01:08:39.235095+0000","last_peered":"2024-09-16T01:08:39.235095+0000","last_clean":"2024-09-16T01:08:39.235095+0000","last_became_active":"2024-09-16T01:08:28.719299+0000","last_became_peered":"2024-09-16T01:08:28.719299+0000","last_unstale":"2024-09-16T01:08:39.235095+0000","last_undegraded":"2024-09-16T01:08:39.235095+0000","last_fullsized":"2024-09-16T01:08:39.235095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:51.616867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770340+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:08:38.770340+0000","last_peered":"2024-09-16T01:08:38.770340+0000","last_clean":"2024-09-16T01:08:38.770340+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:08:38.770340+0000","last_undegraded":"2024-09-16T01:08:38.770340+0000","last_fullsized":"2024-09-16T01:08:38.770340+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770195+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:08:38.770195+0000","last_peered":"2024-09-16T01:08:38.770195+0000","last_clean":"2024-09-16T01:08:38.770195+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:08:38.770195+0000","last_undegraded":"2024-09-16T01:08:38.770195+0000","last_fullsized":"2024-09-16T01:08:38.770195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235648+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:08:39.235648+0000","last_peered":"2024-09-16T01:08:39.235648+0000","last_clean":"2024-09-16T01:08:39.235648+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:08:39.235648+0000","last_undegraded":"2024-09-16T01:08:39.235648+0000","last_fullsized":"2024-09-16T01:08:39.235648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235722+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:08:39.235722+0000","last_peered":"2024-09-16T01:08:39.235722+0000","last_clean":"2024-09-16T01:08:39.235722+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:08:39.235722+0000","last_undegraded":"2024-09-16T01:08:39.235722+0000","last_fullsized":"2024-09-16T01:08:39.235722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235643+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:08:39.235643+0000","last_peered":"2024-09-16T01:08:39.235643+0000","last_clean":"2024-09-16T01:08:39.235643+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:08:39.235643+0000","last_undegraded":"2024-09-16T01:08:39.235643+0000","last_fullsized":"2024-09-16T01:08:39.235643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770667+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:08:38.770667+0000","last_peered":"2024-09-16T01:08:38.770667+0000","last_clean":"2024-09-16T01:08:38.770667+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:08:38.770667+0000","last_undegraded":"2024-09-16T01:08:38.770667+0000","last_fullsized":"2024-09-16T01:08:38.770667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235180+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:08:39.235180+0000","last_peered":"2024-09-16T01:08:39.235180+0000","last_clean":"2024-09-16T01:08:39.235180+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:08:39.235180+0000","last_undegraded":"2024-09-16T01:08:39.235180+0000","last_fullsized":"2024-09-16T01:08:39.235180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.234995+0000","last_change":"2024-09-16T01:08:28.728815+0000","last_active":"2024-09-16T01:08:39.234995+0000","last_peered":"2024-09-16T01:08:39.234995+0000","last_clean":"2024-09-16T01:08:39.234995+0000","last_became_active":"2024-09-16T01:08:28.726726+0000","last_became_peered":"2024-09-16T01:08:28.726726+0000","last_unstale":"2024-09-16T01:08:39.234995+0000","last_undegraded":"2024-09-16T01:08:39.234995+0000","last_fullsized":"2024-09-16T01:08:39.234995+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:57.131619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235748+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:08:39.235748+0000","last_peered":"2024-09-16T01:08:39.235748+0000","last_clean":"2024-09-16T01:08:39.235748+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:08:39.235748+0000","last_undegraded":"2024-09-16T01:08:39.235748+0000","last_fullsized":"2024-09-16T01:08:39.235748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103449+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:08:39.103449+0000","last_peered":"2024-09-16T01:08:39.103449+0000","last_clean":"2024-09-16T01:08:39.103449+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:08:39.103449+0000","last_undegraded":"2024-09-16T01:08:39.103449+0000","last_fullsized":"2024-09-16T01:08:39.103449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236454+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:08:39.236454+0000","last_peered":"2024-09-16T01:08:39.236454+0000","last_clean":"2024-09-16T01:08:39.236454+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:08:39.236454+0000","last_undegraded":"2024-09-16T01:08:39.236454+0000","last_fullsized":"2024-09-16T01:08:39.236454+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235495+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:08:39.235495+0000","last_peered":"2024-09-16T01:08:39.235495+0000","last_clean":"2024-09-16T01:08:39.235495+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:08:39.235495+0000","last_undegraded":"2024-09-16T01:08:39.235495+0000","last_fullsized":"2024-09-16T01:08:39.235495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770812+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:08:38.770812+0000","last_peered":"2024-09-16T01:08:38.770812+0000","last_clean":"2024-09-16T01:08:38.770812+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:08:38.770812+0000","last_undegraded":"2024-09-16T01:08:38.770812+0000","last_fullsized":"2024-09-16T01:08:38.770812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236905+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:08:39.236905+0000","last_peered":"2024-09-16T01:08:39.236905+0000","last_clean":"2024-09-16T01:08:39.236905+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:08:39.236905+0000","last_undegraded":"2024-09-16T01:08:39.236905+0000","last_fullsized":"2024-09-16T01:08:39.236905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235972+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:08:39.235972+0000","last_peered":"2024-09-16T01:08:39.235972+0000","last_clean":"2024-09-16T01:08:39.235972+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:08:39.235972+0000","last_undegraded":"2024-09-16T01:08:39.235972+0000","last_fullsized":"2024-09-16T01:08:39.235972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.102886+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:08:39.102886+0000","last_peered":"2024-09-16T01:08:39.102886+0000","last_clean":"2024-09-16T01:08:39.102886+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:08:39.102886+0000","last_undegraded":"2024-09-16T01:08:39.102886+0000","last_fullsized":"2024-09-16T01:08:39.102886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235070+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:08:39.235070+0000","last_peered":"2024-09-16T01:08:39.235070+0000","last_clean":"2024-09-16T01:08:39.235070+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:08:39.235070+0000","last_undegraded":"2024-09-16T01:08:39.235070+0000","last_fullsized":"2024-09-16T01:08:39.235070+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235293+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:08:39.235293+0000","last_peered":"2024-09-16T01:08:39.235293+0000","last_clean":"2024-09-16T01:08:39.235293+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:08:39.235293+0000","last_undegraded":"2024-09-16T01:08:39.235293+0000","last_fullsized":"2024-09-16T01:08:39.235293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236366+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:08:39.236366+0000","last_peered":"2024-09-16T01:08:39.236366+0000","last_clean":"2024-09-16T01:08:39.236366+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:08:39.236366+0000","last_undegraded":"2024-09-16T01:08:39.236366+0000","last_fullsized":"2024-09-16T01:08:39.236366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236703+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:08:39.236703+0000","last_peered":"2024-09-16T01:08:39.236703+0000","last_clean":"2024-09-16T01:08:39.236703+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:08:39.236703+0000","last_undegraded":"2024-09-16T01:08:39.236703+0000","last_fullsized":"2024-09-16T01:08:39.236703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235754+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:08:39.235754+0000","last_peered":"2024-09-16T01:08:39.235754+0000","last_clean":"2024-09-16T01:08:39.235754+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:08:39.235754+0000","last_undegraded":"2024-09-16T01:08:39.235754+0000","last_fullsized":"2024-09-16T01:08:39.235754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.773062+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:08:38.773062+0000","last_peered":"2024-09-16T01:08:38.773062+0000","last_clean":"2024-09-16T01:08:38.773062+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:08:38.773062+0000","last_undegraded":"2024-09-16T01:08:38.773062+0000","last_fullsized":"2024-09-16T01:08:38.773062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103578+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:08:39.103578+0000","last_peered":"2024-09-16T01:08:39.103578+0000","last_clean":"2024-09-16T01:08:39.103578+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:08:39.103578+0000","last_undegraded":"2024-09-16T01:08:39.103578+0000","last_fullsized":"2024-09-16T01:08:39.103578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235106+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:08:39.235106+0000","last_peered":"2024-09-16T01:08:39.235106+0000","last_clean":"2024-09-16T01:08:39.235106+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:08:39.235106+0000","last_undegraded":"2024-09-16T01:08:39.235106+0000","last_fullsized":"2024-09-16T01:08:39.235106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235176+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:08:39.235176+0000","last_peered":"2024-09-16T01:08:39.235176+0000","last_clean":"2024-09-16T01:08:39.235176+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:08:39.235176+0000","last_undegraded":"2024-09-16T01:08:39.235176+0000","last_fullsized":"2024-09-16T01:08:39.235176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772657+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:08:38.772657+0000","last_peered":"2024-09-16T01:08:38.772657+0000","last_clean":"2024-09-16T01:08:38.772657+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:08:38.772657+0000","last_undegraded":"2024-09-16T01:08:38.772657+0000","last_fullsized":"2024-09-16T01:08:38.772657+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769138+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:08:38.769138+0000","last_peered":"2024-09-16T01:08:38.769138+0000","last_clean":"2024-09-16T01:08:38.769138+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:08:38.769138+0000","last_undegraded":"2024-09-16T01:08:38.769138+0000","last_fullsized":"2024-09-16T01:08:38.769138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236646+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:08:39.236646+0000","last_peered":"2024-09-16T01:08:39.236646+0000","last_clean":"2024-09-16T01:08:39.236646+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:08:39.236646+0000","last_undegraded":"2024-09-16T01:08:39.236646+0000","last_fullsized":"2024-09-16T01:08:39.236646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235727+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:08:39.235727+0000","last_peered":"2024-09-16T01:08:39.235727+0000","last_clean":"2024-09-16T01:08:39.235727+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:08:39.235727+0000","last_undegraded":"2024-09-16T01:08:39.235727+0000","last_fullsized":"2024-09-16T01:08:39.235727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235846+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:08:39.235846+0000","last_peered":"2024-09-16T01:08:39.235846+0000","last_clean":"2024-09-16T01:08:39.235846+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:08:39.235846+0000","last_undegraded":"2024-09-16T01:08:39.235846+0000","last_fullsized":"2024-09-16T01:08:39.235846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235918+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:08:39.235918+0000","last_peered":"2024-09-16T01:08:39.235918+0000","last_clean":"2024-09-16T01:08:39.235918+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:08:39.235918+0000","last_undegraded":"2024-09-16T01:08:39.235918+0000","last_fullsized":"2024-09-16T01:08:39.235918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56411,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":94,"num_read_kb":100,"num_write":159,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":151107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282755,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":489625,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710748,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1120631,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874273,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":493957,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103205,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":485685,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":1085692,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":485037,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":1080887,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953529,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":490080,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":485231,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":529642,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":528292,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":486425,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":43345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":44695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":54379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:09:22.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:09:22.774 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:23 smithi078 ceph-mon[30937]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:23 smithi078 ceph-mon[30937]: pgmap v311: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T01:09:23.178 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:23 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:09:23.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 -- 172.21.15.78:0/110644455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64101420 msgr2=0x7f7b6410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:23.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 --2- 172.21.15.78:0/110644455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64101420 0x7f7b6410d950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009cd0 tx=0x7f7b5c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 -- 172.21.15.78:0/110644455 shutdown_connections 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 --2- 172.21.15.78:0/110644455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64101420 0x7f7b6410d950 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 --2- 172.21.15.78:0/110644455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64100a80 0x7f7b64100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 --2- 172.21.15.78:0/110644455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64106a80 0x7f7b64106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 -- 172.21.15.78:0/110644455 >> 172.21.15.78:0/110644455 conn(0x7f7b640fc7a0 msgr2=0x7f7b640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 -- 172.21.15.78:0/110644455 shutdown_connections 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.256+0000 7f7b6c402640 1 -- 172.21.15.78:0/110644455 wait complete. 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 Processor -- start 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 -- start start 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64100a80 0x7f7b64198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64101420 0x7f7b64198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64110f20 con 0x7f7b64106a80 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64110da0 con 0x7f7b64100a80 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6c402640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641110a0 con 0x7f7b64101420 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50258/0 (socket says 172.21.15.78:50258) 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 -- 172.21.15.78:0/3297969933 learned_addr learned my addr 172.21.15.78:0/3297969933 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a19e640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64100a80 0x7f7b64198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6999d640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64101420 0x7f7b64198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 -- 172.21.15.78:0/3297969933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64101420 msgr2=0x7f7b64198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64101420 0x7f7b64198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 -- 172.21.15.78:0/3297969933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64100a80 msgr2=0x7f7b64198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64100a80 0x7f7b64198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.257+0000 7f7b6a99f640 1 -- 172.21.15.78:0/3297969933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6419d530 con 0x7f7b64106a80 2024-09-16T01:09:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b6a99f640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006010 tx=0x7f7b5c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c004450 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5c003e10 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6419d7c0 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c046480 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.258+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641a5100 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.259+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b64100f60 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.260+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7b5c0045f0 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.260+0000 7f7b5b7fe640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 0x7f7b500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.261+0000 7f7b6a19e640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 0x7f7b500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.261+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7b5c02e030 con 0x7f7b64106a80 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.261+0000 7f7b6a19e640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 0x7f7b500a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009990 tx=0x7f7b60009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:23.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.264+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7b5c04b060 con 0x7f7b64106a80 2024-09-16T01:09:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:23 smithi148 ceph-mon[35030]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:23 smithi148 ceph-mon[35030]: pgmap v311: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T01:09:23.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:23 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:09:23.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.463+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b64062630 con 0x7f7b500a32d0 2024-09-16T01:09:23.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.471+0000 7f7b5b7fe640 1 -- 172.21.15.78:0/3297969933 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340611 (secure 0 0 0) 0x7f7b64062630 con 0x7f7b500a32d0 2024-09-16T01:09:23.475 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:23.476 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 msgr2=0x7f7b500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 0x7f7b500a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009990 tx=0x7f7b60009110 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 msgr2=0x7f7b6419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006010 tx=0x7f7b5c003c20 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 shutdown_connections 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7b500a32d0 0x7f7b500a5790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b64106a80 0x7f7b6419ce50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64101420 0x7f7b64198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 --2- 172.21.15.78:0/3297969933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64100a80 0x7f7b64198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 >> 172.21.15.78:0/3297969933 conn(0x7f7b640fc7a0 msgr2=0x7f7b6406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 shutdown_connections 2024-09-16T01:09:23.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:23.474+0000 7f7b6c402640 1 -- 172.21.15.78:0/3297969933 wait complete. 2024-09-16T01:09:23.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:23 smithi193 ceph-mon[35701]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:23.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:23 smithi193 ceph-mon[35701]: pgmap v311: 129 pgs: 129 active+clean; 632 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-09-16T01:09:23.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:23 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:09:24.483 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-16T01:09:22.004134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647013,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":240,"num_read_kb":219,"num_write":342,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373608,"kb_used_data":11880,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568216,"statfs":{"total":1151940427776,"available":1151557853184,"internally_reserved":0,"allocated":12165120,"data_stored":7783406,"data_compressed":318360,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49191,"num_objects":4,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":88,"num_read_kb":84,"num_write":34,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003252"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103415+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:09:05.103415+0000","last_peered":"2024-09-16T01:09:05.103415+0000","last_clean":"2024-09-16T01:09:05.103415+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:09:05.103415+0000","last_undegraded":"2024-09-16T01:09:05.103415+0000","last_fullsized":"2024-09-16T01:09:05.103415+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235039+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:08:39.235039+0000","last_peered":"2024-09-16T01:08:39.235039+0000","last_clean":"2024-09-16T01:08:39.235039+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:08:39.235039+0000","last_undegraded":"2024-09-16T01:08:39.235039+0000","last_fullsized":"2024-09-16T01:08:39.235039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769984+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:08:38.769984+0000","last_peered":"2024-09-16T01:08:38.769984+0000","last_clean":"2024-09-16T01:08:38.769984+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:08:38.769984+0000","last_undegraded":"2024-09-16T01:08:38.769984+0000","last_fullsized":"2024-09-16T01:08:38.769984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235320+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:08:39.235320+0000","last_peered":"2024-09-16T01:08:39.235320+0000","last_clean":"2024-09-16T01:08:39.235320+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:08:39.235320+0000","last_undegraded":"2024-09-16T01:08:39.235320+0000","last_fullsized":"2024-09-16T01:08:39.235320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236157+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:08:39.236157+0000","last_peered":"2024-09-16T01:08:39.236157+0000","last_clean":"2024-09-16T01:08:39.236157+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:08:39.236157+0000","last_undegraded":"2024-09-16T01:08:39.236157+0000","last_fullsized":"2024-09-16T01:08:39.236157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235231+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:08:39.235231+0000","last_peered":"2024-09-16T01:08:39.235231+0000","last_clean":"2024-09-16T01:08:39.235231+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:08:39.235231+0000","last_undegraded":"2024-09-16T01:08:39.235231+0000","last_fullsized":"2024-09-16T01:08:39.235231+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102254+0000","last_change":"2024-09-16T01:08:26.711638+0000","last_active":"2024-09-16T01:09:05.102254+0000","last_peered":"2024-09-16T01:09:05.102254+0000","last_clean":"2024-09-16T01:09:05.102254+0000","last_became_active":"2024-09-16T01:08:26.711038+0000","last_became_peered":"2024-09-16T01:08:26.711038+0000","last_unstale":"2024-09-16T01:09:05.102254+0000","last_undegraded":"2024-09-16T01:09:05.102254+0000","last_fullsized":"2024-09-16T01:09:05.102254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:12.619264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103270+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:08:39.103270+0000","last_peered":"2024-09-16T01:08:39.103270+0000","last_clean":"2024-09-16T01:08:39.103270+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:08:39.103270+0000","last_undegraded":"2024-09-16T01:08:39.103270+0000","last_fullsized":"2024-09-16T01:08:39.103270+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236249+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:08:39.236249+0000","last_peered":"2024-09-16T01:08:39.236249+0000","last_clean":"2024-09-16T01:08:39.236249+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:08:39.236249+0000","last_undegraded":"2024-09-16T01:08:39.236249+0000","last_fullsized":"2024-09-16T01:08:39.236249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769910+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:08:38.769910+0000","last_peered":"2024-09-16T01:08:38.769910+0000","last_clean":"2024-09-16T01:08:38.769910+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:08:38.769910+0000","last_undegraded":"2024-09-16T01:08:38.769910+0000","last_fullsized":"2024-09-16T01:08:38.769910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236351+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:08:39.236351+0000","last_peered":"2024-09-16T01:08:39.236351+0000","last_clean":"2024-09-16T01:08:39.236351+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:08:39.236351+0000","last_undegraded":"2024-09-16T01:08:39.236351+0000","last_fullsized":"2024-09-16T01:08:39.236351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235876+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:08:39.235876+0000","last_peered":"2024-09-16T01:08:39.235876+0000","last_clean":"2024-09-16T01:08:39.235876+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:08:39.235876+0000","last_undegraded":"2024-09-16T01:08:39.235876+0000","last_fullsized":"2024-09-16T01:08:39.235876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235379+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:08:39.235379+0000","last_peered":"2024-09-16T01:08:39.235379+0000","last_clean":"2024-09-16T01:08:39.235379+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:08:39.235379+0000","last_undegraded":"2024-09-16T01:08:39.235379+0000","last_fullsized":"2024-09-16T01:08:39.235379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.095373+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:09:04.095373+0000","last_peered":"2024-09-16T01:09:04.095373+0000","last_clean":"2024-09-16T01:09:04.095373+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:09:04.095373+0000","last_undegraded":"2024-09-16T01:09:04.095373+0000","last_fullsized":"2024-09-16T01:09:04.095373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236158+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:08:39.236158+0000","last_peered":"2024-09-16T01:08:39.236158+0000","last_clean":"2024-09-16T01:08:39.236158+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:08:39.236158+0000","last_undegraded":"2024-09-16T01:08:39.236158+0000","last_fullsized":"2024-09-16T01:08:39.236158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235421+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:08:39.235421+0000","last_peered":"2024-09-16T01:08:39.235421+0000","last_clean":"2024-09-16T01:08:39.235421+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:08:39.235421+0000","last_undegraded":"2024-09-16T01:08:39.235421+0000","last_fullsized":"2024-09-16T01:08:39.235421+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770368+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:08:38.770368+0000","last_peered":"2024-09-16T01:08:38.770368+0000","last_clean":"2024-09-16T01:08:38.770368+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:08:38.770368+0000","last_undegraded":"2024-09-16T01:08:38.770368+0000","last_fullsized":"2024-09-16T01:08:38.770368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236429+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:08:39.236429+0000","last_peered":"2024-09-16T01:08:39.236429+0000","last_clean":"2024-09-16T01:08:39.236429+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:08:39.236429+0000","last_undegraded":"2024-09-16T01:08:39.236429+0000","last_fullsized":"2024-09-16T01:08:39.236429+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235319+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:08:39.235319+0000","last_peered":"2024-09-16T01:08:39.235319+0000","last_clean":"2024-09-16T01:08:39.235319+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:08:39.235319+0000","last_undegraded":"2024-09-16T01:08:39.235319+0000","last_fullsized":"2024-09-16T01:08:39.235319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094233+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:09:04.094233+0000","last_peered":"2024-09-16T01:09:04.094233+0000","last_clean":"2024-09-16T01:09:04.094233+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:09:04.094233+0000","last_undegraded":"2024-09-16T01:09:04.094233+0000","last_fullsized":"2024-09-16T01:09:04.094233+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236710+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:08:39.236710+0000","last_peered":"2024-09-16T01:08:39.236710+0000","last_clean":"2024-09-16T01:08:39.236710+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:08:39.236710+0000","last_undegraded":"2024-09-16T01:08:39.236710+0000","last_fullsized":"2024-09-16T01:08:39.236710+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769023+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:08:38.769023+0000","last_peered":"2024-09-16T01:08:38.769023+0000","last_clean":"2024-09-16T01:08:38.769023+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:08:38.769023+0000","last_undegraded":"2024-09-16T01:08:38.769023+0000","last_fullsized":"2024-09-16T01:08:38.769023+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094629+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:09:04.094629+0000","last_peered":"2024-09-16T01:09:04.094629+0000","last_clean":"2024-09-16T01:09:04.094629+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:09:04.094629+0000","last_undegraded":"2024-09-16T01:09:04.094629+0000","last_fullsized":"2024-09-16T01:09:04.094629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235926+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:08:39.235926+0000","last_peered":"2024-09-16T01:08:39.235926+0000","last_clean":"2024-09-16T01:08:39.235926+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:08:39.235926+0000","last_undegraded":"2024-09-16T01:08:39.235926+0000","last_fullsized":"2024-09-16T01:08:39.235926+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235938+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:08:39.235938+0000","last_peered":"2024-09-16T01:08:39.235938+0000","last_clean":"2024-09-16T01:08:39.235938+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:08:39.235938+0000","last_undegraded":"2024-09-16T01:08:39.235938+0000","last_fullsized":"2024-09-16T01:08:39.235938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093144+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:09:04.093144+0000","last_peered":"2024-09-16T01:09:04.093144+0000","last_clean":"2024-09-16T01:09:04.093144+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:09:04.093144+0000","last_undegraded":"2024-09-16T01:09:04.093144+0000","last_fullsized":"2024-09-16T01:09:04.093144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770220+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:08:38.770220+0000","last_peered":"2024-09-16T01:08:38.770220+0000","last_clean":"2024-09-16T01:08:38.770220+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:08:38.770220+0000","last_undegraded":"2024-09-16T01:08:38.770220+0000","last_fullsized":"2024-09-16T01:08:38.770220+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235280+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:08:39.235280+0000","last_peered":"2024-09-16T01:08:39.235280+0000","last_clean":"2024-09-16T01:08:39.235280+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:08:39.235280+0000","last_undegraded":"2024-09-16T01:08:39.235280+0000","last_fullsized":"2024-09-16T01:08:39.235280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:15.743094+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:09:15.743094+0000","last_peered":"2024-09-16T01:09:15.743094+0000","last_clean":"2024-09-16T01:09:15.743094+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:09:15.743094+0000","last_undegraded":"2024-09-16T01:09:15.743094+0000","last_fullsized":"2024-09-16T01:09:15.743094+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.234938+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:08:39.234938+0000","last_peered":"2024-09-16T01:08:39.234938+0000","last_clean":"2024-09-16T01:08:39.234938+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:08:39.234938+0000","last_undegraded":"2024-09-16T01:08:39.234938+0000","last_fullsized":"2024-09-16T01:08:39.234938+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236601+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:08:39.236601+0000","last_peered":"2024-09-16T01:08:39.236601+0000","last_clean":"2024-09-16T01:08:39.236601+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:08:39.236601+0000","last_undegraded":"2024-09-16T01:08:39.236601+0000","last_fullsized":"2024-09-16T01:08:39.236601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093321+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:09:04.093321+0000","last_peered":"2024-09-16T01:09:04.093321+0000","last_clean":"2024-09-16T01:09:04.093321+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:09:04.093321+0000","last_undegraded":"2024-09-16T01:09:04.093321+0000","last_fullsized":"2024-09-16T01:09:04.093321+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235229+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:08:39.235229+0000","last_peered":"2024-09-16T01:08:39.235229+0000","last_clean":"2024-09-16T01:08:39.235229+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:08:39.235229+0000","last_undegraded":"2024-09-16T01:08:39.235229+0000","last_fullsized":"2024-09-16T01:08:39.235229+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772709+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:08:38.772709+0000","last_peered":"2024-09-16T01:08:38.772709+0000","last_clean":"2024-09-16T01:08:38.772709+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:08:38.772709+0000","last_undegraded":"2024-09-16T01:08:38.772709+0000","last_fullsized":"2024-09-16T01:08:38.772709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236879+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:08:39.236879+0000","last_peered":"2024-09-16T01:08:39.236879+0000","last_clean":"2024-09-16T01:08:39.236879+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:08:39.236879+0000","last_undegraded":"2024-09-16T01:08:39.236879+0000","last_fullsized":"2024-09-16T01:08:39.236879+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769206+0000","last_change":"2024-09-16T01:08:36.345546+0000","last_active":"2024-09-16T01:08:38.769206+0000","last_peered":"2024-09-16T01:08:38.769206+0000","last_clean":"2024-09-16T01:08:38.769206+0000","last_became_active":"2024-09-16T01:08:36.344943+0000","last_became_peered":"2024-09-16T01:08:36.344943+0000","last_unstale":"2024-09-16T01:08:38.769206+0000","last_undegraded":"2024-09-16T01:08:38.769206+0000","last_fullsized":"2024-09-16T01:08:38.769206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:32.997745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772775+0000","last_change":"2024-09-16T01:08:28.715546+0000","last_active":"2024-09-16T01:08:38.772775+0000","last_peered":"2024-09-16T01:08:38.772775+0000","last_clean":"2024-09-16T01:08:38.772775+0000","last_became_active":"2024-09-16T01:08:28.714380+0000","last_became_peered":"2024-09-16T01:08:28.714380+0000","last_unstale":"2024-09-16T01:08:38.772775+0000","last_undegraded":"2024-09-16T01:08:38.772775+0000","last_fullsized":"2024-09-16T01:08:38.772775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:05.884119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235658+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:08:39.235658+0000","last_peered":"2024-09-16T01:08:39.235658+0000","last_clean":"2024-09-16T01:08:39.235658+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:08:39.235658+0000","last_undegraded":"2024-09-16T01:08:39.235658+0000","last_fullsized":"2024-09-16T01:08:39.235658+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235479+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:08:39.235479+0000","last_peered":"2024-09-16T01:08:39.235479+0000","last_clean":"2024-09-16T01:08:39.235479+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:08:39.235479+0000","last_undegraded":"2024-09-16T01:08:39.235479+0000","last_fullsized":"2024-09-16T01:08:39.235479+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235107+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:08:39.235107+0000","last_peered":"2024-09-16T01:08:39.235107+0000","last_clean":"2024-09-16T01:08:39.235107+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:08:39.235107+0000","last_undegraded":"2024-09-16T01:08:39.235107+0000","last_fullsized":"2024-09-16T01:08:39.235107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:16.131359+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:09:16.131359+0000","last_peered":"2024-09-16T01:09:16.131359+0000","last_clean":"2024-09-16T01:09:16.131359+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:09:16.131359+0000","last_undegraded":"2024-09-16T01:09:16.131359+0000","last_fullsized":"2024-09-16T01:09:16.131359+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236906+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:08:39.236906+0000","last_peered":"2024-09-16T01:08:39.236906+0000","last_clean":"2024-09-16T01:08:39.236906+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:08:39.236906+0000","last_undegraded":"2024-09-16T01:08:39.236906+0000","last_fullsized":"2024-09-16T01:08:39.236906+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235482+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:08:39.235482+0000","last_peered":"2024-09-16T01:08:39.235482+0000","last_clean":"2024-09-16T01:08:39.235482+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:08:39.235482+0000","last_undegraded":"2024-09-16T01:08:39.235482+0000","last_fullsized":"2024-09-16T01:08:39.235482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102908+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:09:05.102908+0000","last_peered":"2024-09-16T01:09:05.102908+0000","last_clean":"2024-09-16T01:09:05.102908+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:09:05.102908+0000","last_undegraded":"2024-09-16T01:09:05.102908+0000","last_fullsized":"2024-09-16T01:09:05.102908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235641+0000","last_change":"2024-09-16T01:08:36.346613+0000","last_active":"2024-09-16T01:08:39.235641+0000","last_peered":"2024-09-16T01:08:39.235641+0000","last_clean":"2024-09-16T01:08:39.235641+0000","last_became_active":"2024-09-16T01:08:36.346205+0000","last_became_peered":"2024-09-16T01:08:36.346205+0000","last_unstale":"2024-09-16T01:08:39.235641+0000","last_undegraded":"2024-09-16T01:08:39.235641+0000","last_fullsized":"2024-09-16T01:08:39.235641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:36.409657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235748+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:08:39.235748+0000","last_peered":"2024-09-16T01:08:39.235748+0000","last_clean":"2024-09-16T01:08:39.235748+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:08:39.235748+0000","last_undegraded":"2024-09-16T01:08:39.235748+0000","last_fullsized":"2024-09-16T01:08:39.235748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:06.111839+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:09:06.111839+0000","last_peered":"2024-09-16T01:09:06.111839+0000","last_clean":"2024-09-16T01:09:06.111839+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:09:06.111839+0000","last_undegraded":"2024-09-16T01:09:06.111839+0000","last_fullsized":"2024-09-16T01:09:06.111839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235851+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:08:39.235851+0000","last_peered":"2024-09-16T01:08:39.235851+0000","last_clean":"2024-09-16T01:08:39.235851+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:08:39.235851+0000","last_undegraded":"2024-09-16T01:08:39.235851+0000","last_fullsized":"2024-09-16T01:08:39.235851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236098+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:08:39.236098+0000","last_peered":"2024-09-16T01:08:39.236098+0000","last_clean":"2024-09-16T01:08:39.236098+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:08:39.236098+0000","last_undegraded":"2024-09-16T01:08:39.236098+0000","last_fullsized":"2024-09-16T01:08:39.236098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235848+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:08:39.235848+0000","last_peered":"2024-09-16T01:08:39.235848+0000","last_clean":"2024-09-16T01:08:39.235848+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:08:39.235848+0000","last_undegraded":"2024-09-16T01:08:39.235848+0000","last_fullsized":"2024-09-16T01:08:39.235848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103616+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:09:05.103616+0000","last_peered":"2024-09-16T01:09:05.103616+0000","last_clean":"2024-09-16T01:09:05.103616+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:09:05.103616+0000","last_undegraded":"2024-09-16T01:09:05.103616+0000","last_fullsized":"2024-09-16T01:09:05.103616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103160+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:08:39.103160+0000","last_peered":"2024-09-16T01:08:39.103160+0000","last_clean":"2024-09-16T01:08:39.103160+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:08:39.103160+0000","last_undegraded":"2024-09-16T01:08:39.103160+0000","last_fullsized":"2024-09-16T01:08:39.103160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235590+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:08:39.235590+0000","last_peered":"2024-09-16T01:08:39.235590+0000","last_clean":"2024-09-16T01:08:39.235590+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:08:39.235590+0000","last_undegraded":"2024-09-16T01:08:39.235590+0000","last_fullsized":"2024-09-16T01:08:39.235590+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236582+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:08:39.236582+0000","last_peered":"2024-09-16T01:08:39.236582+0000","last_clean":"2024-09-16T01:08:39.236582+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:08:39.236582+0000","last_undegraded":"2024-09-16T01:08:39.236582+0000","last_fullsized":"2024-09-16T01:08:39.236582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235554+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:08:39.235554+0000","last_peered":"2024-09-16T01:08:39.235554+0000","last_clean":"2024-09-16T01:08:39.235554+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:08:39.235554+0000","last_undegraded":"2024-09-16T01:08:39.235554+0000","last_fullsized":"2024-09-16T01:08:39.235554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235615+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:08:39.235615+0000","last_peered":"2024-09-16T01:08:39.235615+0000","last_clean":"2024-09-16T01:08:39.235615+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:08:39.235615+0000","last_undegraded":"2024-09-16T01:08:39.235615+0000","last_fullsized":"2024-09-16T01:08:39.235615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.237232+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:08:39.237232+0000","last_peered":"2024-09-16T01:08:39.237232+0000","last_clean":"2024-09-16T01:08:39.237232+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:08:39.237232+0000","last_undegraded":"2024-09-16T01:08:39.237232+0000","last_fullsized":"2024-09-16T01:08:39.237232+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235027+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:08:39.235027+0000","last_peered":"2024-09-16T01:08:39.235027+0000","last_clean":"2024-09-16T01:08:39.235027+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:08:39.235027+0000","last_undegraded":"2024-09-16T01:08:39.235027+0000","last_fullsized":"2024-09-16T01:08:39.235027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235324+0000","last_change":"2024-09-16T01:08:28.726332+0000","last_active":"2024-09-16T01:08:39.235324+0000","last_peered":"2024-09-16T01:08:39.235324+0000","last_clean":"2024-09-16T01:08:39.235324+0000","last_became_active":"2024-09-16T01:08:28.725901+0000","last_became_peered":"2024-09-16T01:08:28.725901+0000","last_unstale":"2024-09-16T01:08:39.235324+0000","last_undegraded":"2024-09-16T01:08:39.235324+0000","last_fullsized":"2024-09-16T01:08:39.235324+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:18:57.922805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:06.111851+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:09:06.111851+0000","last_peered":"2024-09-16T01:09:06.111851+0000","last_clean":"2024-09-16T01:09:06.111851+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:09:06.111851+0000","last_undegraded":"2024-09-16T01:09:06.111851+0000","last_fullsized":"2024-09-16T01:09:06.111851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235033+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:08:39.235033+0000","last_peered":"2024-09-16T01:08:39.235033+0000","last_clean":"2024-09-16T01:08:39.235033+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:08:39.235033+0000","last_undegraded":"2024-09-16T01:08:39.235033+0000","last_fullsized":"2024-09-16T01:08:39.235033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235295+0000","last_change":"2024-09-16T01:08:28.728462+0000","last_active":"2024-09-16T01:08:39.235295+0000","last_peered":"2024-09-16T01:08:39.235295+0000","last_clean":"2024-09-16T01:08:39.235295+0000","last_became_active":"2024-09-16T01:08:28.726704+0000","last_became_peered":"2024-09-16T01:08:28.726704+0000","last_unstale":"2024-09-16T01:08:39.235295+0000","last_undegraded":"2024-09-16T01:08:39.235295+0000","last_fullsized":"2024-09-16T01:08:39.235295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:33:27.278382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.103589+0000","last_change":"2024-09-16T01:08:26.956123+0000","last_active":"2024-09-16T01:09:05.103589+0000","last_peered":"2024-09-16T01:09:05.103589+0000","last_clean":"2024-09-16T01:09:05.103589+0000","last_became_active":"2024-09-16T01:08:26.955746+0000","last_became_peered":"2024-09-16T01:08:26.955746+0000","last_unstale":"2024-09-16T01:09:05.103589+0000","last_undegraded":"2024-09-16T01:09:05.103589+0000","last_fullsized":"2024-09-16T01:09:05.103589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:40.575736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.237265+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:08:39.237265+0000","last_peered":"2024-09-16T01:08:39.237265+0000","last_clean":"2024-09-16T01:08:39.237265+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:08:39.237265+0000","last_undegraded":"2024-09-16T01:08:39.237265+0000","last_fullsized":"2024-09-16T01:08:39.237265+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235392+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:08:39.235392+0000","last_peered":"2024-09-16T01:08:39.235392+0000","last_clean":"2024-09-16T01:08:39.235392+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:08:39.235392+0000","last_undegraded":"2024-09-16T01:08:39.235392+0000","last_fullsized":"2024-09-16T01:08:39.235392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:19.136867+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:09:19.136867+0000","last_peered":"2024-09-16T01:09:19.136867+0000","last_clean":"2024-09-16T01:09:19.136867+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:09:19.136867+0000","last_undegraded":"2024-09-16T01:09:19.136867+0000","last_fullsized":"2024-09-16T01:09:19.136867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236957+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:08:39.236957+0000","last_peered":"2024-09-16T01:08:39.236957+0000","last_clean":"2024-09-16T01:08:39.236957+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:08:39.236957+0000","last_undegraded":"2024-09-16T01:08:39.236957+0000","last_fullsized":"2024-09-16T01:08:39.236957+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235492+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:08:39.235492+0000","last_peered":"2024-09-16T01:08:39.235492+0000","last_clean":"2024-09-16T01:08:39.235492+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:08:39.235492+0000","last_undegraded":"2024-09-16T01:08:39.235492+0000","last_fullsized":"2024-09-16T01:08:39.235492+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.095874+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:09:04.095874+0000","last_peered":"2024-09-16T01:09:04.095874+0000","last_clean":"2024-09-16T01:09:04.095874+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:09:04.095874+0000","last_undegraded":"2024-09-16T01:09:04.095874+0000","last_fullsized":"2024-09-16T01:09:04.095874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235390+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:08:39.235390+0000","last_peered":"2024-09-16T01:08:39.235390+0000","last_clean":"2024-09-16T01:08:39.235390+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:08:39.235390+0000","last_undegraded":"2024-09-16T01:08:39.235390+0000","last_fullsized":"2024-09-16T01:08:39.235390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235981+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:08:39.235981+0000","last_peered":"2024-09-16T01:08:39.235981+0000","last_clean":"2024-09-16T01:08:39.235981+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:08:39.235981+0000","last_undegraded":"2024-09-16T01:08:39.235981+0000","last_fullsized":"2024-09-16T01:08:39.235981+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102595+0000","last_change":"2024-09-16T01:08:26.710927+0000","last_active":"2024-09-16T01:09:05.102595+0000","last_peered":"2024-09-16T01:09:05.102595+0000","last_clean":"2024-09-16T01:09:05.102595+0000","last_became_active":"2024-09-16T01:08:26.709796+0000","last_became_peered":"2024-09-16T01:08:26.709796+0000","last_unstale":"2024-09-16T01:09:05.102595+0000","last_undegraded":"2024-09-16T01:09:05.102595+0000","last_fullsized":"2024-09-16T01:09:05.102595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:30.270120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236238+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:08:39.236238+0000","last_peered":"2024-09-16T01:08:39.236238+0000","last_clean":"2024-09-16T01:08:39.236238+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:08:39.236238+0000","last_undegraded":"2024-09-16T01:08:39.236238+0000","last_fullsized":"2024-09-16T01:08:39.236238+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769443+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:08:38.769443+0000","last_peered":"2024-09-16T01:08:38.769443+0000","last_clean":"2024-09-16T01:08:38.769443+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:08:38.769443+0000","last_undegraded":"2024-09-16T01:08:38.769443+0000","last_fullsized":"2024-09-16T01:08:38.769443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:17.779582+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:09:17.779582+0000","last_peered":"2024-09-16T01:09:17.779582+0000","last_clean":"2024-09-16T01:09:17.779582+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:09:17.779582+0000","last_undegraded":"2024-09-16T01:09:17.779582+0000","last_fullsized":"2024-09-16T01:09:17.779582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52408,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":57,"num_write":30,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236446+0000","last_change":"2024-09-16T01:08:35.768204+0000","last_active":"2024-09-16T01:08:39.236446+0000","last_peered":"2024-09-16T01:08:39.236446+0000","last_clean":"2024-09-16T01:08:39.236446+0000","last_became_active":"2024-09-16T01:08:35.767845+0000","last_became_peered":"2024-09-16T01:08:35.767845+0000","last_unstale":"2024-09-16T01:08:39.236446+0000","last_undegraded":"2024-09-16T01:08:39.236446+0000","last_fullsized":"2024-09-16T01:08:39.236446+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:20.646352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103420+0000","last_change":"2024-09-16T01:08:28.725923+0000","last_active":"2024-09-16T01:08:39.103420+0000","last_peered":"2024-09-16T01:08:39.103420+0000","last_clean":"2024-09-16T01:08:39.103420+0000","last_became_active":"2024-09-16T01:08:28.725595+0000","last_became_peered":"2024-09-16T01:08:28.725595+0000","last_unstale":"2024-09-16T01:08:39.103420+0000","last_undegraded":"2024-09-16T01:08:39.103420+0000","last_fullsized":"2024-09-16T01:08:39.103420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:22.000211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.093415+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:09:04.093415+0000","last_peered":"2024-09-16T01:09:04.093415+0000","last_clean":"2024-09-16T01:09:04.093415+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:09:04.093415+0000","last_undegraded":"2024-09-16T01:09:04.093415+0000","last_fullsized":"2024-09-16T01:09:04.093415+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236524+0000","last_change":"2024-09-16T01:08:35.768360+0000","last_active":"2024-09-16T01:08:39.236524+0000","last_peered":"2024-09-16T01:08:39.236524+0000","last_clean":"2024-09-16T01:08:39.236524+0000","last_became_active":"2024-09-16T01:08:35.767916+0000","last_became_peered":"2024-09-16T01:08:35.767916+0000","last_unstale":"2024-09-16T01:08:39.236524+0000","last_undegraded":"2024-09-16T01:08:39.236524+0000","last_fullsized":"2024-09-16T01:08:39.236524+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:57.329387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236122+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:08:39.236122+0000","last_peered":"2024-09-16T01:08:39.236122+0000","last_clean":"2024-09-16T01:08:39.236122+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:08:39.236122+0000","last_undegraded":"2024-09-16T01:08:39.236122+0000","last_fullsized":"2024-09-16T01:08:39.236122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:04.094940+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:09:04.094940+0000","last_peered":"2024-09-16T01:09:04.094940+0000","last_clean":"2024-09-16T01:09:04.094940+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:09:04.094940+0000","last_undegraded":"2024-09-16T01:09:04.094940+0000","last_fullsized":"2024-09-16T01:09:04.094940+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236917+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:08:39.236917+0000","last_peered":"2024-09-16T01:08:39.236917+0000","last_clean":"2024-09-16T01:08:39.236917+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:08:39.236917+0000","last_undegraded":"2024-09-16T01:08:39.236917+0000","last_fullsized":"2024-09-16T01:08:39.236917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103442+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:08:39.103442+0000","last_peered":"2024-09-16T01:08:39.103442+0000","last_clean":"2024-09-16T01:08:39.103442+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:08:39.103442+0000","last_undegraded":"2024-09-16T01:08:39.103442+0000","last_fullsized":"2024-09-16T01:08:39.103442+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:16.131934+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:09:16.131934+0000","last_peered":"2024-09-16T01:09:16.131934+0000","last_clean":"2024-09-16T01:09:16.131934+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:09:16.131934+0000","last_undegraded":"2024-09-16T01:09:16.131934+0000","last_fullsized":"2024-09-16T01:09:16.131934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235618+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:08:39.235618+0000","last_peered":"2024-09-16T01:08:39.235618+0000","last_clean":"2024-09-16T01:08:39.235618+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:08:39.235618+0000","last_undegraded":"2024-09-16T01:08:39.235618+0000","last_fullsized":"2024-09-16T01:08:39.235618+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235903+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:08:39.235903+0000","last_peered":"2024-09-16T01:08:39.235903+0000","last_clean":"2024-09-16T01:08:39.235903+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:08:39.235903+0000","last_undegraded":"2024-09-16T01:08:39.235903+0000","last_fullsized":"2024-09-16T01:08:39.235903+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236428+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:08:39.236428+0000","last_peered":"2024-09-16T01:08:39.236428+0000","last_clean":"2024-09-16T01:08:39.236428+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:08:39.236428+0000","last_undegraded":"2024-09-16T01:08:39.236428+0000","last_fullsized":"2024-09-16T01:08:39.236428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235323+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:08:39.235323+0000","last_peered":"2024-09-16T01:08:39.235323+0000","last_clean":"2024-09-16T01:08:39.235323+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:08:39.235323+0000","last_undegraded":"2024-09-16T01:08:39.235323+0000","last_fullsized":"2024-09-16T01:08:39.235323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235729+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:08:39.235729+0000","last_peered":"2024-09-16T01:08:39.235729+0000","last_clean":"2024-09-16T01:08:39.235729+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:08:39.235729+0000","last_undegraded":"2024-09-16T01:08:39.235729+0000","last_fullsized":"2024-09-16T01:08:39.235729+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235195+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:08:39.235195+0000","last_peered":"2024-09-16T01:08:39.235195+0000","last_clean":"2024-09-16T01:08:39.235195+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:08:39.235195+0000","last_undegraded":"2024-09-16T01:08:39.235195+0000","last_fullsized":"2024-09-16T01:08:39.235195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102231+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:09:05.102231+0000","last_peered":"2024-09-16T01:09:05.102231+0000","last_clean":"2024-09-16T01:09:05.102231+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:09:05.102231+0000","last_undegraded":"2024-09-16T01:09:05.102231+0000","last_fullsized":"2024-09-16T01:09:05.102231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:05.102278+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:09:05.102278+0000","last_peered":"2024-09-16T01:09:05.102278+0000","last_clean":"2024-09-16T01:09:05.102278+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:09:05.102278+0000","last_undegraded":"2024-09-16T01:09:05.102278+0000","last_fullsized":"2024-09-16T01:09:05.102278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235768+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:08:39.235768+0000","last_peered":"2024-09-16T01:08:39.235768+0000","last_clean":"2024-09-16T01:08:39.235768+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:08:39.235768+0000","last_undegraded":"2024-09-16T01:08:39.235768+0000","last_fullsized":"2024-09-16T01:08:39.235768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770560+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:08:38.770560+0000","last_peered":"2024-09-16T01:08:38.770560+0000","last_clean":"2024-09-16T01:08:38.770560+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:08:38.770560+0000","last_undegraded":"2024-09-16T01:08:38.770560+0000","last_fullsized":"2024-09-16T01:08:38.770560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:09:20.138898+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:09:20.138898+0000","last_peered":"2024-09-16T01:09:20.138898+0000","last_clean":"2024-09-16T01:09:20.138898+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:09:20.138898+0000","last_undegraded":"2024-09-16T01:09:20.138898+0000","last_fullsized":"2024-09-16T01:09:20.138898+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769178+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:08:38.769178+0000","last_peered":"2024-09-16T01:08:38.769178+0000","last_clean":"2024-09-16T01:08:38.769178+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:08:38.769178+0000","last_undegraded":"2024-09-16T01:08:38.769178+0000","last_fullsized":"2024-09-16T01:08:38.769178+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770664+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:08:38.770664+0000","last_peered":"2024-09-16T01:08:38.770664+0000","last_clean":"2024-09-16T01:08:38.770664+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:08:38.770664+0000","last_undegraded":"2024-09-16T01:08:38.770664+0000","last_fullsized":"2024-09-16T01:08:38.770664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235095+0000","last_change":"2024-09-16T01:08:28.719916+0000","last_active":"2024-09-16T01:08:39.235095+0000","last_peered":"2024-09-16T01:08:39.235095+0000","last_clean":"2024-09-16T01:08:39.235095+0000","last_became_active":"2024-09-16T01:08:28.719299+0000","last_became_peered":"2024-09-16T01:08:28.719299+0000","last_unstale":"2024-09-16T01:08:39.235095+0000","last_undegraded":"2024-09-16T01:08:39.235095+0000","last_fullsized":"2024-09-16T01:08:39.235095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:51.616867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770340+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:08:38.770340+0000","last_peered":"2024-09-16T01:08:38.770340+0000","last_clean":"2024-09-16T01:08:38.770340+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:08:38.770340+0000","last_undegraded":"2024-09-16T01:08:38.770340+0000","last_fullsized":"2024-09-16T01:08:38.770340+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770195+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:08:38.770195+0000","last_peered":"2024-09-16T01:08:38.770195+0000","last_clean":"2024-09-16T01:08:38.770195+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:08:38.770195+0000","last_undegraded":"2024-09-16T01:08:38.770195+0000","last_fullsized":"2024-09-16T01:08:38.770195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235648+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:08:39.235648+0000","last_peered":"2024-09-16T01:08:39.235648+0000","last_clean":"2024-09-16T01:08:39.235648+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:08:39.235648+0000","last_undegraded":"2024-09-16T01:08:39.235648+0000","last_fullsized":"2024-09-16T01:08:39.235648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235722+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:08:39.235722+0000","last_peered":"2024-09-16T01:08:39.235722+0000","last_clean":"2024-09-16T01:08:39.235722+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:08:39.235722+0000","last_undegraded":"2024-09-16T01:08:39.235722+0000","last_fullsized":"2024-09-16T01:08:39.235722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235643+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:08:39.235643+0000","last_peered":"2024-09-16T01:08:39.235643+0000","last_clean":"2024-09-16T01:08:39.235643+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:08:39.235643+0000","last_undegraded":"2024-09-16T01:08:39.235643+0000","last_fullsized":"2024-09-16T01:08:39.235643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770667+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:08:38.770667+0000","last_peered":"2024-09-16T01:08:38.770667+0000","last_clean":"2024-09-16T01:08:38.770667+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:08:38.770667+0000","last_undegraded":"2024-09-16T01:08:38.770667+0000","last_fullsized":"2024-09-16T01:08:38.770667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235180+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:08:39.235180+0000","last_peered":"2024-09-16T01:08:39.235180+0000","last_clean":"2024-09-16T01:08:39.235180+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:08:39.235180+0000","last_undegraded":"2024-09-16T01:08:39.235180+0000","last_fullsized":"2024-09-16T01:08:39.235180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.234995+0000","last_change":"2024-09-16T01:08:28.728815+0000","last_active":"2024-09-16T01:08:39.234995+0000","last_peered":"2024-09-16T01:08:39.234995+0000","last_clean":"2024-09-16T01:08:39.234995+0000","last_became_active":"2024-09-16T01:08:28.726726+0000","last_became_peered":"2024-09-16T01:08:28.726726+0000","last_unstale":"2024-09-16T01:08:39.234995+0000","last_undegraded":"2024-09-16T01:08:39.234995+0000","last_fullsized":"2024-09-16T01:08:39.234995+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:57.131619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235748+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:08:39.235748+0000","last_peered":"2024-09-16T01:08:39.235748+0000","last_clean":"2024-09-16T01:08:39.235748+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:08:39.235748+0000","last_undegraded":"2024-09-16T01:08:39.235748+0000","last_fullsized":"2024-09-16T01:08:39.235748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103449+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:08:39.103449+0000","last_peered":"2024-09-16T01:08:39.103449+0000","last_clean":"2024-09-16T01:08:39.103449+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:08:39.103449+0000","last_undegraded":"2024-09-16T01:08:39.103449+0000","last_fullsized":"2024-09-16T01:08:39.103449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236454+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:08:39.236454+0000","last_peered":"2024-09-16T01:08:39.236454+0000","last_clean":"2024-09-16T01:08:39.236454+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:08:39.236454+0000","last_undegraded":"2024-09-16T01:08:39.236454+0000","last_fullsized":"2024-09-16T01:08:39.236454+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235495+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:08:39.235495+0000","last_peered":"2024-09-16T01:08:39.235495+0000","last_clean":"2024-09-16T01:08:39.235495+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:08:39.235495+0000","last_undegraded":"2024-09-16T01:08:39.235495+0000","last_fullsized":"2024-09-16T01:08:39.235495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.770812+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:08:38.770812+0000","last_peered":"2024-09-16T01:08:38.770812+0000","last_clean":"2024-09-16T01:08:38.770812+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:08:38.770812+0000","last_undegraded":"2024-09-16T01:08:38.770812+0000","last_fullsized":"2024-09-16T01:08:38.770812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236905+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:08:39.236905+0000","last_peered":"2024-09-16T01:08:39.236905+0000","last_clean":"2024-09-16T01:08:39.236905+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:08:39.236905+0000","last_undegraded":"2024-09-16T01:08:39.236905+0000","last_fullsized":"2024-09-16T01:08:39.236905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235972+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:08:39.235972+0000","last_peered":"2024-09-16T01:08:39.235972+0000","last_clean":"2024-09-16T01:08:39.235972+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:08:39.235972+0000","last_undegraded":"2024-09-16T01:08:39.235972+0000","last_fullsized":"2024-09-16T01:08:39.235972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.102886+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:08:39.102886+0000","last_peered":"2024-09-16T01:08:39.102886+0000","last_clean":"2024-09-16T01:08:39.102886+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:08:39.102886+0000","last_undegraded":"2024-09-16T01:08:39.102886+0000","last_fullsized":"2024-09-16T01:08:39.102886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235070+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:08:39.235070+0000","last_peered":"2024-09-16T01:08:39.235070+0000","last_clean":"2024-09-16T01:08:39.235070+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:08:39.235070+0000","last_undegraded":"2024-09-16T01:08:39.235070+0000","last_fullsized":"2024-09-16T01:08:39.235070+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235293+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:08:39.235293+0000","last_peered":"2024-09-16T01:08:39.235293+0000","last_clean":"2024-09-16T01:08:39.235293+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:08:39.235293+0000","last_undegraded":"2024-09-16T01:08:39.235293+0000","last_fullsized":"2024-09-16T01:08:39.235293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236366+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:08:39.236366+0000","last_peered":"2024-09-16T01:08:39.236366+0000","last_clean":"2024-09-16T01:08:39.236366+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:08:39.236366+0000","last_undegraded":"2024-09-16T01:08:39.236366+0000","last_fullsized":"2024-09-16T01:08:39.236366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236703+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:08:39.236703+0000","last_peered":"2024-09-16T01:08:39.236703+0000","last_clean":"2024-09-16T01:08:39.236703+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:08:39.236703+0000","last_undegraded":"2024-09-16T01:08:39.236703+0000","last_fullsized":"2024-09-16T01:08:39.236703+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235754+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:08:39.235754+0000","last_peered":"2024-09-16T01:08:39.235754+0000","last_clean":"2024-09-16T01:08:39.235754+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:08:39.235754+0000","last_undegraded":"2024-09-16T01:08:39.235754+0000","last_fullsized":"2024-09-16T01:08:39.235754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.773062+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:08:38.773062+0000","last_peered":"2024-09-16T01:08:38.773062+0000","last_clean":"2024-09-16T01:08:38.773062+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:08:38.773062+0000","last_undegraded":"2024-09-16T01:08:38.773062+0000","last_fullsized":"2024-09-16T01:08:38.773062+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.103578+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:08:39.103578+0000","last_peered":"2024-09-16T01:08:39.103578+0000","last_clean":"2024-09-16T01:08:39.103578+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:08:39.103578+0000","last_undegraded":"2024-09-16T01:08:39.103578+0000","last_fullsized":"2024-09-16T01:08:39.103578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235106+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:08:39.235106+0000","last_peered":"2024-09-16T01:08:39.235106+0000","last_clean":"2024-09-16T01:08:39.235106+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:08:39.235106+0000","last_undegraded":"2024-09-16T01:08:39.235106+0000","last_fullsized":"2024-09-16T01:08:39.235106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235176+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:08:39.235176+0000","last_peered":"2024-09-16T01:08:39.235176+0000","last_clean":"2024-09-16T01:08:39.235176+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:08:39.235176+0000","last_undegraded":"2024-09-16T01:08:39.235176+0000","last_fullsized":"2024-09-16T01:08:39.235176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.772657+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:08:38.772657+0000","last_peered":"2024-09-16T01:08:38.772657+0000","last_clean":"2024-09-16T01:08:38.772657+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:08:38.772657+0000","last_undegraded":"2024-09-16T01:08:38.772657+0000","last_fullsized":"2024-09-16T01:08:38.772657+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:38.769138+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:08:38.769138+0000","last_peered":"2024-09-16T01:08:38.769138+0000","last_clean":"2024-09-16T01:08:38.769138+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:08:38.769138+0000","last_undegraded":"2024-09-16T01:08:38.769138+0000","last_fullsized":"2024-09-16T01:08:38.769138+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.236646+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:08:39.236646+0000","last_peered":"2024-09-16T01:08:39.236646+0000","last_clean":"2024-09-16T01:08:39.236646+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:08:39.236646+0000","last_undegraded":"2024-09-16T01:08:39.236646+0000","last_fullsized":"2024-09-16T01:08:39.236646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235727+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:08:39.235727+0000","last_peered":"2024-09-16T01:08:39.235727+0000","last_clean":"2024-09-16T01:08:39.235727+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:08:39.235727+0000","last_undegraded":"2024-09-16T01:08:39.235727+0000","last_fullsized":"2024-09-16T01:08:39.235727+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235846+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:08:39.235846+0000","last_peered":"2024-09-16T01:08:39.235846+0000","last_clean":"2024-09-16T01:08:39.235846+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:08:39.235846+0000","last_undegraded":"2024-09-16T01:08:39.235846+0000","last_fullsized":"2024-09-16T01:08:39.235846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T01:08:39.235918+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:08:39.235918+0000","last_peered":"2024-09-16T01:08:39.235918+0000","last_clean":"2024-09-16T01:08:39.235918+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:08:39.235918+0000","last_undegraded":"2024-09-16T01:08:39.235918+0000","last_fullsized":"2024-09-16T01:08:39.235918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":56411,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":104,"num_read_kb":100,"num_write":159,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":169233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282755,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":489625,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710748,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1120631,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874273,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":493957,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103206,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":489781,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":1085692,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":485037,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":1080887,"data_compressed":53506,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":490080,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":485231,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":538705,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":537355,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":486425,"data_compressed":17538,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":52408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":54379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:09:24.490 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T01:09:24.491 INFO:tasks.cephadm:Setting up client nodes... 2024-09-16T01:09:24.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-16T01:09:24.827 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:25.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:25.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-09-16T01:09:25.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.254 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:25 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 -- 172.21.15.78:0/3636137171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 msgr2=0x7fd24810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:25.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 --2- 172.21.15.78:0/3636137171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd24810a050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd240009ca0 tx=0x7fd24002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 -- 172.21.15.78:0/3636137171 shutdown_connections 2024-09-16T01:09:25.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 --2- 172.21.15.78:0/3636137171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd24810a050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 --2- 172.21.15.78:0/3636137171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248102e00 0x7fd248103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 --2- 172.21.15.78:0/3636137171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248101c00 0x7fd248102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 -- 172.21.15.78:0/3636137171 >> 172.21.15.78:0/3636137171 conn(0x7fd2480fd390 msgr2=0x7fd2480ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 -- 172.21.15.78:0/3636137171 shutdown_connections 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.339+0000 7fd2502c1640 1 -- 172.21.15.78:0/3636137171 wait complete. 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 Processor -- start 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 -- start start 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 0x7fd24819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 0x7fd24819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810d860 con 0x7fd2481037c0 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810d6e0 con 0x7fd248102e00 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd2502c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd24810d9e0 con 0x7fd248101c00 2024-09-16T01:09:25.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60916/0 (socket says 172.21.15.78:60916) 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24d85c640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 0x7fd24819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 -- 172.21.15.78:0/654363600 learned_addr learned my addr 172.21.15.78:0/654363600 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e05d640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 0x7fd24819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 -- 172.21.15.78:0/654363600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 msgr2=0x7fd24819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 0x7fd24819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 -- 172.21.15.78:0/654363600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 msgr2=0x7fd24819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 0x7fd24819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e85e640 1 -- 172.21.15.78:0/654363600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2481a1cb0 con 0x7fd2481037c0 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24e05d640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 0x7fd24819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.340+0000 7fd24d85c640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 0x7fd24819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd24e85e640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd240005e30 tx=0x7fd240031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:25.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240003c30 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd240003dd0 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481a1f40 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24002fde0 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.341+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2481a23f0 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.342+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2481a2650 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.343+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd240004050 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.344+0000 7fd23f7fe640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 0x7fd2340a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.344+0000 7fd24e05d640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 0x7fd2340a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.344+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd2400f10e0 con 0x7fd2481037c0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.344+0000 7fd24e05d640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 0x7fd2340a5840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd248102c60 tx=0x7fd24400a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:25.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.347+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2400bdcc0 con 0x7fd2481037c0 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:25 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: pgmap v312: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 10 op/s 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:25 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:25.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.597+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd2481a28c0 con 0x7fd2481037c0 2024-09-16T01:09:25.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.599+0000 7fd23f7fe640 1 -- 172.21.15.78:0/654363600 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fd2400bd5c0 con 0x7fd2481037c0 2024-09-16T01:09:25.600 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-09-16T01:09:25.600 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBFhedmCiuwIxAAHXgsXz51HzSTIVOqRMJ64w== 2024-09-16T01:09:25.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 msgr2=0x7fd2340a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:25.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 0x7fd2340a5840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd248102c60 tx=0x7fd24400a560 comp rx=0 tx=0).stop 2024-09-16T01:09:25.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 msgr2=0x7fd2481a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd240005e30 tx=0x7fd240031ee0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 shutdown_connections 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd2340a3380 0x7fd2340a5840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2481037c0 0x7fd2481a1640 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd248102e00 0x7fd24819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 --2- 172.21.15.78:0/654363600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd248101c00 0x7fd24819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.601+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 >> 172.21.15.78:0/654363600 conn(0x7fd2480fd390 msgr2=0x7fd248074420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.602+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 shutdown_connections 2024-09-16T01:09:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:25.602+0000 7fd2502c1640 1 -- 172.21.15.78:0/654363600 wait complete. 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:09:26.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:26 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.368 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:09:26.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-16T01:09:26.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T01:09:26.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T01:09:26.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T01:09:26.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:09:26.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:09:26.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:26 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.404 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T01:09:26.404 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-16T01:09:26.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/654363600' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:09:26.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:09:26.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:26 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:09:26.688 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- 172.21.15.78:0/1167018598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 msgr2=0x7f82c0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 --2- 172.21.15.78:0/1167018598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0103110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009cd0 tx=0x7f82b002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- 172.21.15.78:0/1167018598 shutdown_connections 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 --2- 172.21.15.78:0/1167018598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 --2- 172.21.15.78:0/1167018598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 --2- 172.21.15.78:0/1167018598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0103110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- 172.21.15.78:0/1167018598 >> 172.21.15.78:0/1167018598 conn(0x7f82c00fc7d0 msgr2=0x7f82c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- 172.21.15.78:0/1167018598 shutdown_connections 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- 172.21.15.78:0/1167018598 wait complete. 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 Processor -- start 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.200+0000 7f82c621e640 1 -- start start 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:27.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01120a0 con 0x7f82c0102d30 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0111f20 con 0x7f82c01036e0 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0112220 con 0x7f82c01082a0 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60926/0 (socket says 172.21.15.78:60926) 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 -- 172.21.15.78:0/3183794827 learned_addr learned my addr 172.21.15.78:0/3183794827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bffff640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82beffd640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 -- 172.21.15.78:0/3183794827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 msgr2=0x7f82c019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 -- 172.21.15.78:0/3183794827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 msgr2=0x7f82c0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:27.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 -- 172.21.15.78:0/3183794827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c019d4d0 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bffff640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82beffd640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bf7fe640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82b002ae00 tx=0x7f82b0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0004040 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82b00046c0 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c019d760 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.201+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0046480 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.202+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c01a50a0 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.202+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c0103110 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.203+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f82b00305a0 con 0x7f82c0102d30 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.204+0000 7f82bcff9640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 0x7f82ac0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.204+0000 7f82beffd640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 0x7f82ac0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:27.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.204+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f82b002e030 con 0x7f82c0102d30 2024-09-16T01:09:27.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.205+0000 7f82beffd640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 0x7f82ac0a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82c00fedf0 tx=0x7f82a8005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:27.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.208+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f82b004b060 con 0x7f82c0102d30 2024-09-16T01:09:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:27 smithi148 ceph-mon[35030]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-16T01:09:27.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.421+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f82c01a5270 con 0x7f82c0102d30 2024-09-16T01:09:27.422 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:27 smithi078 ceph-mon[30937]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-16T01:09:27.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.426+0000 7f82bcff9640 1 -- 172.21.15.78:0/3183794827 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264049 (secure 0 0 0) 0x7f82b0030890 con 0x7f82c0102d30 2024-09-16T01:09:27.430 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:27.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 msgr2=0x7f82ac0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:27.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 0x7f82ac0a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f82c00fedf0 tx=0x7f82a8005e50 comp rx=0 tx=0).stop 2024-09-16T01:09:27.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 msgr2=0x7f82c0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82b002ae00 tx=0x7f82b0003970 comp rx=0 tx=0).stop 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 shutdown_connections 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82ac0a32d0 0x7f82ac0a5790 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82c01082a0 0x7f82c019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82c01036e0 0x7f82c0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 --2- 172.21.15.78:0/3183794827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82c0102d30 0x7f82c0198520 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 >> 172.21.15.78:0/3183794827 conn(0x7f82c00fc7d0 msgr2=0x7f82c0105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 shutdown_connections 2024-09-16T01:09:27.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:27.429+0000 7f82c621e640 1 -- 172.21.15.78:0/3183794827 wait complete. 2024-09-16T01:09:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:27 smithi193 ceph-mon[35701]: pgmap v313: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-16T01:09:28.125 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:28 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3183794827' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T01:09:28.156 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":1698663383},{"type":"v1","addr":"172.21.15.78:6801","nonce":1698663383}]},"active_addr":"172.21.15.78:6801/1698663383","active_change":"2024-09-16T01:01:37.912281+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":557237582}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1671709190}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1816200719}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3382939699}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2774297562}]}]} 2024-09-16T01:09:28.162 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-16T01:09:28.162 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-16T01:09:28.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:09:28.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:28 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3183794827' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T01:09:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:28.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:28 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3183794827' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-16T01:09:28.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 -- 172.21.15.78:0/4037485111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0100e20 msgr2=0x7f3ac0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/4037485111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0100e20 0x7f3ac0101200 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009900 tx=0x7f3ab402a480 comp rx=0 tx=0).stop 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 -- 172.21.15.78:0/4037485111 shutdown_connections 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/4037485111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0102ef0 0x7f3ac010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/4037485111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102570 0x7f3ac01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/4037485111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0100e20 0x7f3ac0101200 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 -- 172.21.15.78:0/4037485111 >> 172.21.15.78:0/4037485111 conn(0x7f3ac00fc7d0 msgr2=0x7f3ac00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 -- 172.21.15.78:0/4037485111 shutdown_connections 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 -- 172.21.15.78:0/4037485111 wait complete. 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.899+0000 7f3ac5e72640 1 Processor -- start 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 -- start start 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0100e20 0x7f3ac019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:28.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102ef0 0x7f3ac019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0110fa0 con 0x7f3ac0102570 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0110e20 con 0x7f3ac0100e20 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3ac5e72640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0111120 con 0x7f3ac0102ef0 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60948/0 (socket says 172.21.15.78:60948) 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 -- 172.21.15.78:0/406700338 learned_addr learned my addr 172.21.15.78:0/406700338 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abf7fe640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0100e20 0x7f3ac019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abffff640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102ef0 0x7f3ac019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 -- 172.21.15.78:0/406700338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102ef0 msgr2=0x7f3ac019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:28.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102ef0 0x7f3ac019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 -- 172.21.15.78:0/406700338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0100e20 msgr2=0x7f3ac019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0100e20 0x7f3ac019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.900+0000 7f3abeffd640 1 -- 172.21.15.78:0/406700338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac019f6c0 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3abeffd640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000b4a0 tx=0x7f3ab000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0014070 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab0010af0 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0017d10 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac019f9b0 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.901+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac01a7290 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.902+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac0101200 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.903+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3ab0017070 con 0x7f3ac0102570 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.903+0000 7f3abcff9640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 0x7f3aa80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:28.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.904+0000 7f3abf7fe640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 0x7f3aa80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.904+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3ab00ceef0 con 0x7f3ac0102570 2024-09-16T01:09:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.904+0000 7f3abf7fe640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 0x7f3aa80a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4000c00 tx=0x7f3ab40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:28.908+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ab009b330 con 0x7f3ac0102570 2024-09-16T01:09:29.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.074+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ac01a7520 con 0x7f3ac0102570 2024-09-16T01:09:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.076+0000 7f3abcff9640 1 -- 172.21.15.78:0/406700338 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7f3ab009f630 con 0x7f3ac0102570 2024-09-16T01:09:29.077 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:29.078 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:09:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 msgr2=0x7f3aa80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 0x7f3aa80a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4000c00 tx=0x7f3ab40058d0 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 msgr2=0x7f3ac019ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000b4a0 tx=0x7f3ab000b960 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 shutdown_connections 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f3aa80a32d0 0x7f3aa80a5790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3ac0102ef0 0x7f3ac019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ac0102570 0x7f3ac019ac50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 --2- 172.21.15.78:0/406700338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ac0100e20 0x7f3ac019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 >> 172.21.15.78:0/406700338 conn(0x7f3ac00fc7d0 msgr2=0x7f3ac006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.078+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 shutdown_connections 2024-09-16T01:09:29.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:29.079+0000 7f3ac5e72640 1 -- 172.21.15.78:0/406700338 wait complete. 2024-09-16T01:09:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:29 smithi148 ceph-mon[35030]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-16T01:09:29.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:29 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/406700338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:29.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:29 smithi078 ceph-mon[30937]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-16T01:09:29.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:29 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/406700338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:29 smithi193 ceph-mon[35701]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-16T01:09:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:29 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/406700338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:29.828 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-16T01:09:29.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:09:30.074 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 -- 172.21.15.78:0/2792681414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc106c30 msgr2=0x7f0ddc107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 --2- 172.21.15.78:0/2792681414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc106c30 0x7f0ddc107010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009ca0 tx=0x7f0dcc02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 -- 172.21.15.78:0/2792681414 shutdown_connections 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 --2- 172.21.15.78:0/2792681414 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc101b40 0x7f0ddc10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 --2- 172.21.15.78:0/2792681414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 --2- 172.21.15.78:0/2792681414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc106c30 0x7f0ddc107010 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.563+0000 7f0de46ac640 1 -- 172.21.15.78:0/2792681414 >> 172.21.15.78:0/2792681414 conn(0x7f0ddc077210 msgr2=0x7f0ddc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:30.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- 172.21.15.78:0/2792681414 shutdown_connections 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- 172.21.15.78:0/2792681414 wait complete. 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 Processor -- start 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- start start 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc101b40 0x7f0ddc199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc106c30 0x7f0ddc1a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc111040 con 0x7f0ddc1011a0 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc110ec0 con 0x7f0ddc106c30 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de46ac640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc1111c0 con 0x7f0ddc101b40 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de2448640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de2448640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60962/0 (socket says 172.21.15.78:60962) 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.564+0000 7f0de2448640 1 -- 172.21.15.78:0/3536803651 learned_addr learned my addr 172.21.15.78:0/3536803651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:30.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2c49640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc106c30 0x7f0ddc1a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de1c47640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc101b40 0x7f0ddc199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 -- 172.21.15.78:0/3536803651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc101b40 msgr2=0x7f0ddc199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc101b40 0x7f0ddc199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 -- 172.21.15.78:0/3536803651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc106c30 msgr2=0x7f0ddc1a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc106c30 0x7f0ddc1a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 -- 172.21.15.78:0/3536803651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc1a4a90 con 0x7f0ddc1011a0 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de2448640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0097b0 tx=0x7f0dcc031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:30.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc02fba0 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ddc1a4c60 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dcc02fd40 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc03b930 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.565+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ddc1a5110 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.566+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ddc101600 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.567+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0dcc030cc0 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.568+0000 7f0ddb7fe640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 0x7f0dc40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.568+0000 7f0de1c47640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 0x7f0dc40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.568+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0dcc0f1370 con 0x7f0ddc1011a0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.568+0000 7f0de1c47640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 0x7f0dc40a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc100be0 tx=0x7f0dd0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:30.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.572+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0dcc0bde00 con 0x7f0ddc1011a0 2024-09-16T01:09:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.740+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ddc1090a0 con 0x7f0ddc1011a0 2024-09-16T01:09:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.742+0000 7f0ddb7fe640 1 -- 172.21.15.78:0/3536803651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24175 (secure 0 0 0) 0x7f0dcc0bd700 con 0x7f0ddc1011a0 2024-09-16T01:09:30.743 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:09:30.744 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:08:38.437722+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:09:30.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 msgr2=0x7f0dc40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:30.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 0x7f0dc40a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc100be0 tx=0x7f0dd0008040 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 msgr2=0x7f0ddc1994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc0097b0 tx=0x7f0dcc031b10 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 shutdown_connections 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0dc40a32d0 0x7f0dc40a5790 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ddc106c30 0x7f0ddc1a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ddc101b40 0x7f0ddc199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 --2- 172.21.15.78:0/3536803651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ddc1011a0 0x7f0ddc1994d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 >> 172.21.15.78:0/3536803651 conn(0x7f0ddc077210 msgr2=0x7f0ddc103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 shutdown_connections 2024-09-16T01:09:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:30.744+0000 7f0de46ac640 1 -- 172.21.15.78:0/3536803651 wait complete. 2024-09-16T01:09:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:31 smithi148 ceph-mon[35030]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-09-16T01:09:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:31 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3536803651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:31.425 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:31 smithi078 ceph-mon[30937]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-09-16T01:09:31.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:31 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3536803651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:31.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-16T01:09:31.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-16T01:09:31.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-16T01:09:31.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-16T01:09:31.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-16T01:09:31.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-16T01:09:31.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-16T01:09:31.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-16T01:09:31.460 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-16T01:09:31.460 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-16T01:09:31.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-16T01:09:31.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-16T01:09:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:31 smithi193 ceph-mon[35701]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-09-16T01:09:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:31 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3536803651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:09:32.066 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.416 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.429 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.432 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:32.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- 172.21.15.78:0/4130349042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720106a80 msgr2=0x7f9720106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 --2- 172.21.15.78:0/4130349042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720106a80 0x7f9720106e60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f971c009930 tx=0x7f971c02a420 comp rx=0 tx=0).stop 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- 172.21.15.78:0/4130349042 shutdown_connections 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 --2- 172.21.15.78:0/4130349042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720101420 0x7f972010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 --2- 172.21.15.78:0/4130349042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9720100a80 0x7f9720100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 --2- 172.21.15.78:0/4130349042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720106a80 0x7f9720106e60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- 172.21.15.78:0/4130349042 >> 172.21.15.78:0/4130349042 conn(0x7f97200fc7a0 msgr2=0x7f97200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- 172.21.15.78:0/4130349042 shutdown_connections 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- 172.21.15.78:0/4130349042 wait complete. 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 Processor -- start 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.629+0000 7f972871e640 1 -- start start 2024-09-16T01:09:32.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 0x7f97201984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9720101420 0x7f97201989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720106a80 0x7f972019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9720110ec0 con 0x7f9720100a80 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9720110d40 con 0x7f9720101420 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9720111040 con 0x7f9720106a80 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 0x7f97201984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 0x7f97201984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60982/0 (socket says 172.21.15.78:60982) 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 -- 172.21.15.78:0/2940161894 learned_addr learned my addr 172.21.15.78:0/2940161894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f9726cbb640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720106a80 0x7f972019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 -- 172.21.15.78:0/2940161894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720106a80 msgr2=0x7f972019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720106a80 0x7f972019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 -- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9720101420 msgr2=0x7f97201989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9720101420 0x7f97201989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972019d460 con 0x7f9720100a80 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f9726cbb640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9720106a80 0x7f972019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97264ba640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 0x7f97201984b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f971c009230 tx=0x7f971c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c03f030 con 0x7f9720100a80 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972019d6f0 con 0x7f9720100a80 2024-09-16T01:09:32.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971c003ad0 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c035070 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.630+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97201a5030 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.631+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9720062980 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f971c042030 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f97177fe640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f970c0a32d0 0x7f970c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f9725cb9640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f970c0a32d0 0x7f970c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f971c0f06d0 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f97177fe640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] conn(0x7f970c0ae000 0x7f970c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.633+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f970c0b0b10 con 0x7f970c0ae000 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.634+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f971c0f0b20 con 0x7f9720100a80 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.634+0000 7f9725cb9640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f970c0a32d0 0x7f970c0a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9720105150 tx=0x7f9710006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.634+0000 7f9726cbb640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] conn(0x7f970c0ae000 0x7f970c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:32.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.634+0000 7f9726cbb640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] conn(0x7f970c0ae000 0x7f970c0b0440 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.635+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== osd.1 v2:172.21.15.148:6800/593162683 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f970c0b0b10 con 0x7f970c0ae000 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.648+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 --> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9720062d80 con 0x7f970c0ae000 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f97177fe640 1 -- 172.21.15.78:0/2940161894 <== osd.1 v2:172.21.15.148:6800/593162683 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9720062d80 con 0x7f970c0ae000 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] conn(0x7f970c0ae000 msgr2=0x7f970c0b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] conn(0x7f970c0ae000 0x7f970c0b0440 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f970c0a32d0 msgr2=0x7f970c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f970c0a32d0 0x7f970c0a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9720105150 tx=0x7f9710006c90 comp rx=0 tx=0).stop 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 msgr2=0x7f97201984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 --2- 172.21.15.78:0/2940161894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9720100a80 0x7f97201984b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f971c009230 tx=0x7f971c004270 comp rx=0 tx=0).stop 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f9726cbb640 1 -- 172.21.15.78:0/2940161894 reap_dead start 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 shutdown_connections 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 >> 172.21.15.78:0/2940161894 conn(0x7f97200fc7a0 msgr2=0x7f972006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:32.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 shutdown_connections 2024-09-16T01:09:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:32.649+0000 7f972871e640 1 -- 172.21.15.78:0/2940161894 wait complete. 2024-09-16T01:09:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:33 smithi078 ceph-mon[30937]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-09-16T01:09:33.328 INFO:teuthology.orchestra.run.smithi078.stdout:55834574925 2024-09-16T01:09:33.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-16T01:09:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:33 smithi148 ceph-mon[35030]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-09-16T01:09:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:33 smithi193 ceph-mon[35701]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-09-16T01:09:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- 172.21.15.78:0/1292512674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc41027a0 msgr2=0x7f5cc41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/1292512674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc41027a0 0x7f5cc41067f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009900 tx=0x7f5cb802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- 172.21.15.78:0/1292512674 shutdown_connections 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/1292512674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5cc4106f20 0x7f5cc410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/1292512674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc41027a0 0x7f5cc41067f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/1292512674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc4101df0 0x7f5cc41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- 172.21.15.78:0/1292512674 >> 172.21.15.78:0/1292512674 conn(0x7f5cc40fd8b0 msgr2=0x7f5cc40ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- 172.21.15.78:0/1292512674 shutdown_connections 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- 172.21.15.78:0/1292512674 wait complete. 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 Processor -- start 2024-09-16T01:09:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.315+0000 7f5cca3eb640 1 -- start start 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5cc4101df0 0x7f5cc40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc41027a0 0x7f5cc4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 0x7f5cc4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4111fa0 con 0x7f5cc4106f20 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4111e20 con 0x7f5cc4101df0 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cca3eb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4112120 con 0x7f5cc41027a0 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 0x7f5cc4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 0x7f5cc4074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55548/0 (socket says 172.21.15.78:55548) 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 -- 172.21.15.78:0/311071853 learned_addr learned my addr 172.21.15.78:0/311071853 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc37fe640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc41027a0 0x7f5cc4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc3fff640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5cc4101df0 0x7f5cc40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 -- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc41027a0 msgr2=0x7f5cc4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc41027a0 0x7f5cc4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 -- 172.21.15.78:0/311071853 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5cc4101df0 msgr2=0x7f5cc40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5cc4101df0 0x7f5cc40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc41a6910 con 0x7f5cc4106f20 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc37fe640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cc41027a0 0x7f5cc4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.316+0000 7f5cc8988640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 0x7f5cc4074f70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00cce0 tx=0x7f5cbc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00f450 con 0x7f5cc4106f20 2024-09-16T01:09:34.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cbc00fad0 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc41a6c00 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc018c10 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc41a7140 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.317+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5cc4062880 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.319+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5cbc00f5f0 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.319+0000 7f5cc17fa640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5cac0a3350 0x7f5cac0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc3fff640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5cac0a3350 0x7f5cac0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5cbc0d0640 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc17fa640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] conn(0x7f5cac0ae080 0x7f5cac0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cbc0047f0 con 0x7f5cac0ae080 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f5cbc0d0a90 con 0x7f5cc4106f20 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc3fff640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5cac0a3350 0x7f5cac0a5810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009890 tx=0x7f5cb4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc37fe640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] conn(0x7f5cac0ae080 0x7f5cac0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.320+0000 7f5cc37fe640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] conn(0x7f5cac0ae080 0x7f5cac0b04c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.321+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== osd.8 v2:172.21.15.193:6816/692862816 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cbc0047f0 con 0x7f5cac0ae080 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.335+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 --> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5cc4062d00 con 0x7f5cac0ae080 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.335+0000 7f5cc17fa640 1 -- 172.21.15.78:0/311071853 <== osd.8 v2:172.21.15.193:6816/692862816 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5cc4062d00 con 0x7f5cac0ae080 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] conn(0x7f5cac0ae080 msgr2=0x7f5cac0b04c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] conn(0x7f5cac0ae080 0x7f5cac0b04c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5cac0a3350 msgr2=0x7f5cac0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5cac0a3350 0x7f5cac0a5810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009890 tx=0x7f5cb4008040 comp rx=0 tx=0).stop 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 msgr2=0x7f5cc4074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 --2- 172.21.15.78:0/311071853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cc4106f20 0x7f5cc4074f70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00cce0 tx=0x7f5cbc00a620 comp rx=0 tx=0).stop 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cc8988640 1 -- 172.21.15.78:0/311071853 reap_dead start 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 shutdown_connections 2024-09-16T01:09:34.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 >> 172.21.15.78:0/311071853 conn(0x7f5cc40fd8b0 msgr2=0x7f5cc40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:34.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 shutdown_connections 2024-09-16T01:09:34.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.336+0000 7f5cca3eb640 1 -- 172.21.15.78:0/311071853 wait complete. 2024-09-16T01:09:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 -- 172.21.15.78:0/3433736003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0105a90 msgr2=0x7fccd0105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 --2- 172.21.15.78:0/3433736003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0105a90 0x7fccd0105e70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009900 tx=0x7fccc002a410 comp rx=0 tx=0).stop 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 -- 172.21.15.78:0/3433736003 shutdown_connections 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 --2- 172.21.15.78:0/3433736003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101bc0 0x7fccd010d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 --2- 172.21.15.78:0/3433736003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0101240 0x7fccd0101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 --2- 172.21.15.78:0/3433736003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0105a90 0x7fccd0105e70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 -- 172.21.15.78:0/3433736003 >> 172.21.15.78:0/3433736003 conn(0x7fccd0077210 msgr2=0x7fccd00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 -- 172.21.15.78:0/3433736003 shutdown_connections 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.879+0000 7fccd6248640 1 -- 172.21.15.78:0/3433736003 wait complete. 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 Processor -- start 2024-09-16T01:09:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 -- start start 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101240 0x7fccd010e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 0x7fccd010ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0105a90 0x7fccd010f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0112060 con 0x7fccd0101bc0 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0111ee0 con 0x7fccd0105a90 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccd6248640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01121e0 con 0x7fccd0101240 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 0x7fccd010ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 0x7fccd010ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55554/0 (socket says 172.21.15.78:55554) 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fccceffd640 1 -- 172.21.15.78:0/1840563769 learned_addr learned my addr 172.21.15.78:0/1840563769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.880+0000 7fcccffff640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0105a90 0x7fccd010f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fcccf7fe640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101240 0x7fccd010e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 -- 172.21.15.78:0/1840563769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101240 msgr2=0x7fccd010e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101240 0x7fccd010e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 -- 172.21.15.78:0/1840563769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0105a90 msgr2=0x7fccd010f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0105a90 0x7fccd010f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd01a6920 con 0x7fccd0101bc0 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fcccf7fe640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccd0101240 0x7fccd010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fcccffff640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccd0105a90 0x7fccd010f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccceffd640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 0x7fccd010ec50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c920 tx=0x7fccc400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4007c70 con 0x7fccd0101bc0 2024-09-16T01:09:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc4007e10 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4018890 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd01a6c10 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.881+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd01a7150 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.882+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fccd0104330 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.883+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fccc4018a30 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fccccff9640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fccb80a32d0 0x7fccb80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fcccf7fe640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fccb80a32d0 0x7fccb80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fccc40d02f0 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fccccff9640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] conn(0x7fccb80ae000 0x7fccb80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccb80b0b10 con 0x7fccb80ae000 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fccc40d0740 con 0x7fccd0101bc0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fcccffff640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] conn(0x7fccb80ae000 0x7fccb80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fcccf7fe640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fccb80a32d0 0x7fccb80a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000c00 tx=0x7fccc00041e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.884+0000 7fcccffff640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] conn(0x7fccb80ae000 0x7fccb80b0440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.885+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== osd.0 v2:172.21.15.78:6802/3172670933 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccb80b0b10 con 0x7fccb80ae000 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.898+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 --> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fccd0102250 con 0x7fccb80ae000 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccccff9640 1 -- 172.21.15.78:0/1840563769 <== osd.0 v2:172.21.15.78:6802/3172670933 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fccd0102250 con 0x7fccb80ae000 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] conn(0x7fccb80ae000 msgr2=0x7fccb80b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] conn(0x7fccb80ae000 0x7fccb80b0440 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fccb80a32d0 msgr2=0x7fccb80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fccb80a32d0 0x7fccb80a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000c00 tx=0x7fccc00041e0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 msgr2=0x7fccd010ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 --2- 172.21.15.78:0/1840563769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd0101bc0 0x7fccd010ec50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c920 tx=0x7fccc400cde0 comp rx=0 tx=0).stop 2024-09-16T01:09:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fcccffff640 1 -- 172.21.15.78:0/1840563769 reap_dead start 2024-09-16T01:09:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 shutdown_connections 2024-09-16T01:09:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 >> 172.21.15.78:0/1840563769 conn(0x7fccd0077210 msgr2=0x7fccd0103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 shutdown_connections 2024-09-16T01:09:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:34.899+0000 7fccd6248640 1 -- 172.21.15.78:0/1840563769 wait complete. 2024-09-16T01:09:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 -- 172.21.15.78:0/522876147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48102790 msgr2=0x7f7a481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 --2- 172.21.15.78:0/522876147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48102790 0x7f7a481067e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009900 tx=0x7f7a3c02a410 comp rx=0 tx=0).stop 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 -- 172.21.15.78:0/522876147 shutdown_connections 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 --2- 172.21.15.78:0/522876147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a48106f10 0x7f7a4810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 --2- 172.21.15.78:0/522876147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48102790 0x7f7a481067e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 --2- 172.21.15.78:0/522876147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a48101de0 0x7f7a481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 -- 172.21.15.78:0/522876147 >> 172.21.15.78:0/522876147 conn(0x7f7a480fd8c0 msgr2=0x7f7a480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 -- 172.21.15.78:0/522876147 shutdown_connections 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.084+0000 7f7a4e367640 1 -- 172.21.15.78:0/522876147 wait complete. 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 Processor -- start 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 -- start start 2024-09-16T01:09:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 0x7f7a481a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a48102790 0x7f7a481a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a48106f10 0x7f7a4810dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7a48111f70 con 0x7f7a48101de0 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a48111df0 con 0x7f7a48102790 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4e367640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a481120f0 con 0x7f7a48106f10 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 0x7f7a481a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 0x7f7a481a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55576/0 (socket says 172.21.15.78:55576) 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a47fff640 1 -- 172.21.15.78:0/3648925561 learned_addr learned my addr 172.21.15.78:0/3648925561 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.085+0000 7f7a4c904640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a48106f10 0x7f7a4810dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 -- 172.21.15.78:0/3648925561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a48106f10 msgr2=0x7f7a4810dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a48106f10 0x7f7a4810dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 -- 172.21.15.78:0/3648925561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a48102790 msgr2=0x7f7a481a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a48102790 0x7f7a481a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4810e420 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a47fff640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 0x7f7a481a4830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800d770 tx=0x7f7a3800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38017070 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4810e710 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a38005500 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38004e00 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.086+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a481ab760 con 0x7f7a48101de0 2024-09-16T01:09:35.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.087+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a48062940 con 0x7f7a48101de0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.088+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7a38004fa0 con 0x7f7a48101de0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a457fa640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7a300a32d0 0x7f7a300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a477fe640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7a300a32d0 0x7f7a300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7a380cfc00 con 0x7f7a48101de0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a457fa640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] conn(0x7f7a300ae000 0x7f7a300b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a38001d30 con 0x7f7a300ae000 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7a380d0050 con 0x7f7a48101de0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a4c904640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] conn(0x7f7a300ae000 0x7f7a300b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a477fe640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7a300a32d0 0x7f7a300a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02a930 tx=0x7f7a3c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.089+0000 7f7a4c904640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] conn(0x7f7a300ae000 0x7f7a300b0440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.090+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== osd.3 v2:172.21.15.78:6810/419734598 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7a38001d30 con 0x7f7a300ae000 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.104+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 --> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7a48062dc0 con 0x7f7a300ae000 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a457fa640 1 -- 172.21.15.78:0/3648925561 <== osd.3 v2:172.21.15.78:6810/419734598 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7a48062dc0 con 0x7f7a300ae000 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] conn(0x7f7a300ae000 msgr2=0x7f7a300b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] conn(0x7f7a300ae000 0x7f7a300b0440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7a300a32d0 msgr2=0x7f7a300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7a300a32d0 0x7f7a300a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02a930 tx=0x7f7a3c000bc0 comp rx=0 tx=0).stop 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 msgr2=0x7f7a481a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 --2- 172.21.15.78:0/3648925561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a48101de0 0x7f7a481a4830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800d770 tx=0x7f7a3800dc30 comp rx=0 tx=0).stop 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4c904640 1 -- 172.21.15.78:0/3648925561 reap_dead start 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 shutdown_connections 2024-09-16T01:09:35.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 >> 172.21.15.78:0/3648925561 conn(0x7f7a480fd8c0 msgr2=0x7f7a480fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:35.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 shutdown_connections 2024-09-16T01:09:35.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:35.105+0000 7f7a4e367640 1 -- 172.21.15.78:0/3648925561 wait complete. 2024-09-16T01:09:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:35 smithi148 ceph-mon[35030]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-16T01:09:35.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:35 smithi078 ceph-mon[30937]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-16T01:09:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:35 smithi193 ceph-mon[35701]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-09-16T01:09:35.820 INFO:teuthology.orchestra.run.smithi078.stdout:107374182467 2024-09-16T01:09:35.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 -- 172.21.15.78:0/3381308161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 msgr2=0x7f823010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 --2- 172.21.15.78:0/3381308161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823010e790 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8228009900 tx=0x7f822802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 -- 172.21.15.78:0/3381308161 shutdown_connections 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 --2- 172.21.15.78:0/3381308161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823010e790 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 --2- 172.21.15.78:0/3381308161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82301037a0 0x7f823010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 --2- 172.21.15.78:0/3381308161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8230102df0 0x7f82301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 -- 172.21.15.78:0/3381308161 >> 172.21.15.78:0/3381308161 conn(0x7f82300fc7d0 msgr2=0x7f82300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 -- 172.21.15.78:0/3381308161 shutdown_connections 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.465+0000 7f82376c4640 1 -- 172.21.15.78:0/3381308161 wait complete. 2024-09-16T01:09:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 Processor -- start 2024-09-16T01:09:36.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 -- start start 2024-09-16T01:09:36.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8230102df0 0x7f8230198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:36.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82301037a0 0x7f8230198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f82301120f0 con 0x7f823010c2f0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8230111f70 con 0x7f8230102df0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.466+0000 7f82376c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8230112270 con 0x7f82301037a0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823019cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55592/0 (socket says 172.21.15.78:55592) 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 -- 172.21.15.78:0/357092863 learned_addr learned my addr 172.21.15.78:0/357092863 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235460640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8230102df0 0x7f8230198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8234c5f640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82301037a0 0x7f8230198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 -- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82301037a0 msgr2=0x7f8230198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82301037a0 0x7f8230198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 -- 172.21.15.78:0/357092863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8230102df0 msgr2=0x7f8230198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8230102df0 0x7f8230198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823019d5e0 con 0x7f823010c2f0 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f8235c61640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823019cf00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8228005ab0 tx=0x7f82280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822803f030 con 0x7f823010c2f0 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823019d870 con 0x7f823010c2f0 2024-09-16T01:09:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8228003ce0 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228035070 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.467+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82301a51b0 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.468+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8230062900 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8228042030 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82180a32d0 0x7f82180a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f8235460640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82180a32d0 0x7f82180a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f82280f0720 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] conn(0x7f82180aded0 0x7f82180b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8228001d30 con 0x7f82180aded0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.470+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f82280f0b70 con 0x7f823010c2f0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.471+0000 7f8235460640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82180a32d0 0x7f82180a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8224005fd0 tx=0x7f8224009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.471+0000 7f8234c5f640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] conn(0x7f82180aded0 0x7f82180b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:36.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.471+0000 7f8234c5f640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] conn(0x7f82180aded0 0x7f82180b0310 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.472+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== osd.11 v2:172.21.15.193:6824/1055668594 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8228001d30 con 0x7f82180aded0 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.496+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 --> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8230062d00 con 0x7f82180aded0 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f822e7fc640 1 -- 172.21.15.78:0/357092863 <== osd.11 v2:172.21.15.193:6824/1055668594 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8230062d00 con 0x7f82180aded0 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] conn(0x7f82180aded0 msgr2=0x7f82180b0310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] conn(0x7f82180aded0 0x7f82180b0310 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82180a32d0 msgr2=0x7f82180a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82180a32d0 0x7f82180a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8224005fd0 tx=0x7f8224009680 comp rx=0 tx=0).stop 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 msgr2=0x7f823019cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 --2- 172.21.15.78:0/357092863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823010c2f0 0x7f823019cf00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8228005ab0 tx=0x7f82280042f0 comp rx=0 tx=0).stop 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f8235c61640 1 -- 172.21.15.78:0/357092863 reap_dead start 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 shutdown_connections 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.497+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 >> 172.21.15.78:0/357092863 conn(0x7f82300fc7d0 msgr2=0x7f82300fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:36.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.498+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 shutdown_connections 2024-09-16T01:09:36.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:36.498+0000 7f82376c4640 1 -- 172.21.15.78:0/357092863 wait complete. 2024-09-16T01:09:37.176 INFO:teuthology.orchestra.run.smithi078.stdout:249108103209 2024-09-16T01:09:37.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-16T01:09:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:37 smithi148 ceph-mon[35030]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-16T01:09:37.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:37 smithi078 ceph-mon[30937]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-16T01:09:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:37 smithi193 ceph-mon[35701]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-16T01:09:37.630 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:37.637 INFO:teuthology.orchestra.run.smithi078.stdout:330712481819 2024-09-16T01:09:37.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-16T01:09:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-16T01:09:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-09-16T01:09:38.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:38 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T01:09:38.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-16T01:09:38.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-09-16T01:09:38.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:38 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T01:09:38.457 INFO:teuthology.orchestra.run.smithi078.stdout:34359738450 2024-09-16T01:09:38.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-16T01:09:38.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-09-16T01:09:38.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-09-16T01:09:38.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:38 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-16T01:09:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- 172.21.15.78:0/1425898734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0418101df0 msgr2=0x7f04181021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 --2- 172.21.15.78:0/1425898734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0418101df0 0x7f04181021d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0400009900 tx=0x7f040002a410 comp rx=0 tx=0).stop 2024-09-16T01:09:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- 172.21.15.78:0/1425898734 shutdown_connections 2024-09-16T01:09:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 --2- 172.21.15.78:0/1425898734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0418106f20 0x7f041810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 --2- 172.21.15.78:0/1425898734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04181027a0 0x7f04181067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 --2- 172.21.15.78:0/1425898734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0418101df0 0x7f04181021d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- 172.21.15.78:0/1425898734 >> 172.21.15.78:0/1425898734 conn(0x7f04180fd8f0 msgr2=0x7f04180ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- 172.21.15.78:0/1425898734 shutdown_connections 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- 172.21.15.78:0/1425898734 wait complete. 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 Processor -- start 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.616+0000 7f041ea61640 1 -- start start 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0418101df0 0x7f04180744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 0x7f0418074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0418106f20 0x7f0418074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0418111f80 con 0x7f04181027a0 2024-09-16T01:09:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0418111e00 con 0x7f0418101df0 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041ea61640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0418112100 con 0x7f0418106f20 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 0x7f0418074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 0x7f0418074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55612/0 (socket says 172.21.15.78:55612) 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 -- 172.21.15.78:0/3887695599 learned_addr learned my addr 172.21.15.78:0/3887695599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f041cffe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0418106f20 0x7f0418074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f0417fff640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0418101df0 0x7f04180744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 -- 172.21.15.78:0/3887695599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0418106f20 msgr2=0x7f0418074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0418106f20 0x7f0418074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 -- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0418101df0 msgr2=0x7f04180744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0418101df0 0x7f04180744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f04177fe640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04181a6880 con 0x7f04181027a0 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.617+0000 7f0417fff640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0418101df0 0x7f04180744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f04177fe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 0x7f0418074a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f040800c920 tx=0x7f040800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408007c70 con 0x7f04181027a0 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04181a6b70 con 0x7f04181027a0 2024-09-16T01:09:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0408007e10 con 0x7f04181027a0 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408018890 con 0x7f04181027a0 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.618+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04181a70b0 con 0x7f04181027a0 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.619+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04180628c0 con 0x7f04181027a0 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.620+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0408018a30 con 0x7f04181027a0 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f04157fa640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04040a32d0 0x7f04040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:38.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f0417fff640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04040a32d0 0x7f04040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f04080d0370 con 0x7f04181027a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f04157fa640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] conn(0x7f04040ae4a0 0x7f04040b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04040b1010 con 0x7f04040ae4a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f04080d07c0 con 0x7f04181027a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f0417fff640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04040a32d0 0x7f04040a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0400000c00 tx=0x7f0400032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.621+0000 7f041cffe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] conn(0x7f04040ae4a0 0x7f04040b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.622+0000 7f041cffe640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] conn(0x7f04040ae4a0 0x7f04040b0940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.623+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== osd.7 v2:172.21.15.148:6816/883929363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04040b1010 con 0x7f04040ae4a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.636+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 --> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0418062cc0 con 0x7f04040ae4a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f04157fa640 1 -- 172.21.15.78:0/3887695599 <== osd.7 v2:172.21.15.148:6816/883929363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0418062cc0 con 0x7f04040ae4a0 2024-09-16T01:09:38.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] conn(0x7f04040ae4a0 msgr2=0x7f04040b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] conn(0x7f04040ae4a0 0x7f04040b0940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04040a32d0 msgr2=0x7f04040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f04040a32d0 0x7f04040a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0400000c00 tx=0x7f0400032c70 comp rx=0 tx=0).stop 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 msgr2=0x7f0418074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041ea61640 1 --2- 172.21.15.78:0/3887695599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04181027a0 0x7f0418074a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f040800c920 tx=0x7f040800cde0 comp rx=0 tx=0).stop 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.637+0000 7f041cffe640 1 -- 172.21.15.78:0/3887695599 reap_dead start 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.638+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 shutdown_connections 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.638+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 >> 172.21.15.78:0/3887695599 conn(0x7f04180fd8f0 msgr2=0x7f04180fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.638+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 shutdown_connections 2024-09-16T01:09:38.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:38.638+0000 7f041ea61640 1 -- 172.21.15.78:0/3887695599 wait complete. 2024-09-16T01:09:39.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T01:09:39.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:39 smithi078 ceph-mon[30937]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T01:09:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T01:09:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:39 smithi148 ceph-mon[35030]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T01:09:39.391 INFO:teuthology.orchestra.run.smithi078.stdout:219043332144 2024-09-16T01:09:39.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-16T01:09:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 6 op/s 2024-09-16T01:09:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-09-16T01:09:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-09-16T01:09:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-09-16T01:09:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 8]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-16T01:09:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:39 smithi193 ceph-mon[35701]: osdmap e90: 12 total, 12 up, 12 in 2024-09-16T01:09:39.637 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:39.644 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:39.646 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:39.651 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 -- 172.21.15.78:0/3129049730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c102790 msgr2=0x7f187c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 --2- 172.21.15.78:0/3129049730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c102790 0x7f187c1067e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f186c009900 tx=0x7f186c02a410 comp rx=0 tx=0).stop 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 -- 172.21.15.78:0/3129049730 shutdown_connections 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 --2- 172.21.15.78:0/3129049730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f187c106f10 0x7f187c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 --2- 172.21.15.78:0/3129049730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c102790 0x7f187c1067e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 --2- 172.21.15.78:0/3129049730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c101de0 0x7f187c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 -- 172.21.15.78:0/3129049730 >> 172.21.15.78:0/3129049730 conn(0x7f187c0fd8c0 msgr2=0x7f187c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 -- 172.21.15.78:0/3129049730 shutdown_connections 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.882+0000 7f1883487640 1 -- 172.21.15.78:0/3129049730 wait complete. 2024-09-16T01:09:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 Processor -- start 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 -- start start 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 0x7f187c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c102790 0x7f187c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f187c106f10 0x7f187c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f187c112010 con 0x7f187c101de0 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f187c111e90 con 0x7f187c102790 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1883487640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f187c112190 con 0x7f187c106f10 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1881223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 0x7f187c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1881223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 0x7f187c1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55628/0 (socket says 172.21.15.78:55628) 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.883+0000 7f1881223640 1 -- 172.21.15.78:0/2435495056 learned_addr learned my addr 172.21.15.78:0/2435495056 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1880a22640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c102790 0x7f187c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 -- 172.21.15.78:0/2435495056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f187c106f10 msgr2=0x7f187c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f187c106f10 0x7f187c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 -- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c102790 msgr2=0x7f187c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c102790 0x7f187c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187c19d560 con 0x7f187c101de0 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1880a22640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f187c102790 0x7f187c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1881223640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 0x7f187c1985b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f187800b4a0 tx=0x7f187800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878014070 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1878010af0 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187c19d850 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1878017d10 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.884+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f187c1a5130 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.885+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f187c062980 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.886+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1878017070 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f18727fc640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18680a32d0 0x7f18680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f1880a22640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18680a32d0 0x7f18680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f18780cf510 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f18727fc640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] conn(0x7f18680ae4a0 0x7f18680b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f187801ae50 con 0x7f18680ae4a0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.887+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f18780cf960 con 0x7f187c101de0 2024-09-16T01:09:39.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.888+0000 7f1881a24640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] conn(0x7f18680ae4a0 0x7f18680b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.888+0000 7f1880a22640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18680a32d0 0x7f18680a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f186c000c00 tx=0x7f186c032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.888+0000 7f1881a24640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] conn(0x7f18680ae4a0 0x7f18680b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.889+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== osd.4 v2:172.21.15.148:6808/2237990015 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f187801ae50 con 0x7f18680ae4a0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.902+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 --> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f187c062e00 con 0x7f18680ae4a0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f18727fc640 1 -- 172.21.15.78:0/2435495056 <== osd.4 v2:172.21.15.148:6808/2237990015 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f187c062e00 con 0x7f18680ae4a0 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] conn(0x7f18680ae4a0 msgr2=0x7f18680b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] conn(0x7f18680ae4a0 0x7f18680b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18680a32d0 msgr2=0x7f18680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18680a32d0 0x7f18680a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f186c000c00 tx=0x7f186c032d50 comp rx=0 tx=0).stop 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 msgr2=0x7f187c1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 --2- 172.21.15.78:0/2435495056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f187c101de0 0x7f187c1985b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f187800b4a0 tx=0x7f187800b960 comp rx=0 tx=0).stop 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1881a24640 1 -- 172.21.15.78:0/2435495056 reap_dead start 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 shutdown_connections 2024-09-16T01:09:39.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 >> 172.21.15.78:0/2435495056 conn(0x7f187c0fd8c0 msgr2=0x7f187c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:39.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.903+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 shutdown_connections 2024-09-16T01:09:39.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:39.904+0000 7f1883487640 1 -- 172.21.15.78:0/2435495056 wait complete. 2024-09-16T01:09:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:40.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:40 smithi078 ceph-mon[30937]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T01:09:40.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:40 smithi148 ceph-mon[35030]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T01:09:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:40 smithi193 ceph-mon[35701]: osdmap e91: 12 total, 12 up, 12 in 2024-09-16T01:09:40.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 -- 172.21.15.78:0/2660607709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 msgr2=0x7effb81008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 --2- 172.21.15.78:0/2660607709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb81008e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7effa8009900 tx=0x7effa802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 -- 172.21.15.78:0/2660607709 shutdown_connections 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 --2- 172.21.15.78:0/2660607709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effb8106ed0 0x7effb810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 --2- 172.21.15.78:0/2660607709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8100eb0 0x7effb81067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 --2- 172.21.15.78:0/2660607709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb81008e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 -- 172.21.15.78:0/2660607709 >> 172.21.15.78:0/2660607709 conn(0x7effb80fc5c0 msgr2=0x7effb80fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 -- 172.21.15.78:0/2660607709 shutdown_connections 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.939+0000 7effbcb84640 1 -- 172.21.15.78:0/2660607709 wait complete. 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 Processor -- start 2024-09-16T01:09:40.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 -- start start 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effb8100eb0 0x7effb8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8106ed0 0x7effb819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7effb8111fa0 con 0x7effb8100500 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7effb8111e20 con 0x7effb8100eb0 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effbcb84640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7effb8112120 con 0x7effb8106ed0 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.940+0000 7effb659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb8198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55646/0 (socket says 172.21.15.78:55646) 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 -- 172.21.15.78:0/4089446571 learned_addr learned my addr 172.21.15.78:0/4089446571 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb5d9b640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effb8100eb0 0x7effb8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb6d9d640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8106ed0 0x7effb819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 -- 172.21.15.78:0/4089446571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8106ed0 msgr2=0x7effb819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8106ed0 0x7effb819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 -- 172.21.15.78:0/4089446571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effb8100eb0 msgr2=0x7effb8198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effb8100eb0 0x7effb8198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb819d4d0 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb6d9d640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effb8106ed0 0x7effb819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effb659c640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb8198520 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7effa8005ab0 tx=0x7effa80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa803f030 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa8003ce0 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb819d760 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8035070 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.941+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb81a50a0 con 0x7effb8100500 2024-09-16T01:09:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.942+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7effb8062900 con 0x7effb8100500 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.943+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7effa8042030 con 0x7effb8100500 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effa77fe640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7effa00a3350 0x7effa00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effb5d9b640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7effa00a3350 0x7effa00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7effa80f0a80 con 0x7effb8100500 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effa77fe640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] conn(0x7effa00ae520 0x7effa00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7effa8001d30 con 0x7effa00ae520 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7effa80f0ed0 con 0x7effb8100500 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effb6d9d640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] conn(0x7effa00ae520 0x7effa00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.944+0000 7effb5d9b640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7effa00a3350 0x7effa00a5810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7effb81047b0 tx=0x7effac009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.945+0000 7effb6d9d640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] conn(0x7effa00ae520 0x7effa00b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.945+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== osd.9 v2:172.21.15.78:6826/409695790 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7effa8001d30 con 0x7effa00ae520 2024-09-16T01:09:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.958+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 --> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7effb8062d30 con 0x7effa00ae520 2024-09-16T01:09:40.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effa77fe640 1 -- 172.21.15.78:0/4089446571 <== osd.9 v2:172.21.15.78:6826/409695790 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7effb8062d30 con 0x7effa00ae520 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] conn(0x7effa00ae520 msgr2=0x7effa00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] conn(0x7effa00ae520 0x7effa00b09c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7effa00a3350 msgr2=0x7effa00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7effa00a3350 0x7effa00a5810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7effb81047b0 tx=0x7effac009110 comp rx=0 tx=0).stop 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 msgr2=0x7effb8198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 --2- 172.21.15.78:0/4089446571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effb8100500 0x7effb8198520 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7effa8005ab0 tx=0x7effa80042f0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effb6d9d640 1 -- 172.21.15.78:0/4089446571 reap_dead start 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 shutdown_connections 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 >> 172.21.15.78:0/4089446571 conn(0x7effb80fc5c0 msgr2=0x7effb8105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:41.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 shutdown_connections 2024-09-16T01:09:41.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:40.959+0000 7effbcb84640 1 -- 172.21.15.78:0/4089446571 wait complete. 2024-09-16T01:09:41.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:41 smithi078 ceph-mon[30937]: pgmap v322: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-16T01:09:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:41 smithi148 ceph-mon[35030]: pgmap v322: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-16T01:09:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:41 smithi193 ceph-mon[35701]: pgmap v322: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-16T01:09:41.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 -- 172.21.15.78:0/1457508495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 msgr2=0x7f791c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 --2- 172.21.15.78:0/1457508495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c100ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7910009930 tx=0x7f791002a420 comp rx=0 tx=0).stop 2024-09-16T01:09:41.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 -- 172.21.15.78:0/1457508495 shutdown_connections 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 --2- 172.21.15.78:0/1457508495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f791c101420 0x7f791c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 --2- 172.21.15.78:0/1457508495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c100ee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 --2- 172.21.15.78:0/1457508495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791c106a80 0x7f791c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 -- 172.21.15.78:0/1457508495 >> 172.21.15.78:0/1457508495 conn(0x7f791c0fc7a0 msgr2=0x7f791c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 -- 172.21.15.78:0/1457508495 shutdown_connections 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 -- 172.21.15.78:0/1457508495 wait complete. 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.930+0000 7f792166c640 1 Processor -- start 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 -- start start 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c0fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791c101420 0x7f791c0fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f791c106a80 0x7f791c0fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f791c110e80 con 0x7f791c100a80 2024-09-16T01:09:41.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f791c110d00 con 0x7f791c101420 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f791c111000 con 0x7f791c106a80 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c0fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c0fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55654/0 (socket says 172.21.15.78:55654) 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 -- 172.21.15.78:0/3077002719 learned_addr learned my addr 172.21.15.78:0/3077002719 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791a7fc640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791c101420 0x7f791c0fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791b7fe640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f791c106a80 0x7f791c0fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 -- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f791c106a80 msgr2=0x7f791c0fd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f791c106a80 0x7f791c0fd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 -- 172.21.15.78:0/3077002719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791c101420 msgr2=0x7f791c0fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791c101420 0x7f791c0fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c1a6920 con 0x7f791c100a80 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f791affd640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c0fcd90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f791400db80 tx=0x7f791400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914017070 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7914003b70 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791c1a6c10 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.931+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79140106c0 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.932+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791c1a7120 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.932+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f791c062900 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7914004020 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79040a32d0 0x7f79040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f791a7fc640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79040a32d0 0x7f79040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f79140d0270 con 0x7f791c100a80 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] conn(0x7f79040ae4a0 0x7f79040b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f791401fe50 con 0x7f79040ae4a0 2024-09-16T01:09:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.934+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f79140d06c0 con 0x7f791c100a80 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.935+0000 7f791a7fc640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79040a32d0 0x7f79040a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7910009900 tx=0x7f79100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.935+0000 7f791b7fe640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] conn(0x7f79040ae4a0 0x7f79040b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.935+0000 7f791b7fe640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] conn(0x7f79040ae4a0 0x7f79040b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.936+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== osd.5 v2:172.21.15.193:6808/2281160557 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f791401fe50 con 0x7f79040ae4a0 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.948+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 --> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f791c062d80 con 0x7f79040ae4a0 2024-09-16T01:09:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f7903fff640 1 -- 172.21.15.78:0/3077002719 <== osd.5 v2:172.21.15.193:6808/2281160557 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f791c062d80 con 0x7f79040ae4a0 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] conn(0x7f79040ae4a0 msgr2=0x7f79040b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] conn(0x7f79040ae4a0 0x7f79040b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79040a32d0 msgr2=0x7f79040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79040a32d0 0x7f79040a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7910009900 tx=0x7f79100058d0 comp rx=0 tx=0).stop 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 msgr2=0x7f791c0fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 --2- 172.21.15.78:0/3077002719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f791c100a80 0x7f791c0fcd90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f791400db80 tx=0x7f791400b630 comp rx=0 tx=0).stop 2024-09-16T01:09:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f791b7fe640 1 -- 172.21.15.78:0/3077002719 reap_dead start 2024-09-16T01:09:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 shutdown_connections 2024-09-16T01:09:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 >> 172.21.15.78:0/3077002719 conn(0x7f791c0fc7a0 msgr2=0x7f791c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 shutdown_connections 2024-09-16T01:09:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:41.949+0000 7f792166c640 1 -- 172.21.15.78:0/3077002719 wait complete. 2024-09-16T01:09:42.625 INFO:teuthology.orchestra.run.smithi078.stdout:279172874278 2024-09-16T01:09:42.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-16T01:09:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2831385157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b4102df0 msgr2=0x7fc3b41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2831385157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b4102df0 0x7fc3b41031d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009900 tx=0x7fc3b002a410 comp rx=0 tx=0).stop 2024-09-16T01:09:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2831385157 shutdown_connections 2024-09-16T01:09:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2831385157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b410c2f0 0x7fc3b410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2831385157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3b41037a0 0x7fc3b410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2831385157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b4102df0 0x7fc3b41031d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2831385157 >> 172.21.15.78:0/2831385157 conn(0x7fc3b40fc7d0 msgr2=0x7fc3b40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2831385157 shutdown_connections 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2831385157 wait complete. 2024-09-16T01:09:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 Processor -- start 2024-09-16T01:09:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.788+0000 7fc3bbee0640 1 -- start start 2024-09-16T01:09:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b4102df0 0x7fc3b4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 0x7fc3b4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3b410c2f0 0x7fc3b419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b41120f0 con 0x7fc3b41037a0 2024-09-16T01:09:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b4111f70 con 0x7fc3b4102df0 2024-09-16T01:09:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3bbee0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b4112270 con 0x7fc3b410c2f0 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 0x7fc3b4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 0x7fc3b4198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55674/0 (socket says 172.21.15.78:55674) 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 -- 172.21.15.78:0/2476483441 learned_addr learned my addr 172.21.15.78:0/2476483441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3ba47d640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3b410c2f0 0x7fc3b419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b9c7c640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b4102df0 0x7fc3b4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 -- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3b410c2f0 msgr2=0x7fc3b419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3b410c2f0 0x7fc3b419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 -- 172.21.15.78:0/2476483441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b4102df0 msgr2=0x7fc3b4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b4102df0 0x7fc3b4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b419d5c0 con 0x7fc3b41037a0 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b9c7c640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3b4102df0 0x7fc3b4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3b947b640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 0x7fc3b4198b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800d740 tx=0x7fc3a800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.789+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8017070 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.790+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3a800de50 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.790+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b419d8b0 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.790+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a800f6b0 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.790+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b41a5190 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.791+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3b4062900 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.792+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc3a801a070 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3a6ffd640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc3a00a32d0 0x7fc3a00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3a80cfbf0 con 0x7fc3b41037a0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3b9c7c640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc3a00a32d0 0x7fc3a00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3a6ffd640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] conn(0x7fc3a00ae4a0 0x7fc3a00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3a801de50 con 0x7fc3a00ae4a0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc3a80d0040 con 0x7fc3b41037a0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3b9c7c640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc3a00a32d0 0x7fc3a00a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0000c00 tx=0x7fc3b0032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.793+0000 7fc3ba47d640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] conn(0x7fc3a00ae4a0 0x7fc3a00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.794+0000 7fc3ba47d640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] conn(0x7fc3a00ae4a0 0x7fc3a00b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.795+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== osd.2 v2:172.21.15.193:6800/3453663394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3a801de50 con 0x7fc3a00ae4a0 2024-09-16T01:09:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.808+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 --> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3b4062d80 con 0x7fc3a00ae4a0 2024-09-16T01:09:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.809+0000 7fc3a6ffd640 1 -- 172.21.15.78:0/2476483441 <== osd.2 v2:172.21.15.193:6800/3453663394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc3b4062d80 con 0x7fc3a00ae4a0 2024-09-16T01:09:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.809+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] conn(0x7fc3a00ae4a0 msgr2=0x7fc3a00b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.809+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] conn(0x7fc3a00ae4a0 0x7fc3a00b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc3a00a32d0 msgr2=0x7fc3a00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fc3a00a32d0 0x7fc3a00a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0000c00 tx=0x7fc3b0032c70 comp rx=0 tx=0).stop 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 msgr2=0x7fc3b4198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 --2- 172.21.15.78:0/2476483441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b41037a0 0x7fc3b4198b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800d740 tx=0x7fc3a800dc00 comp rx=0 tx=0).stop 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3ba47d640 1 -- 172.21.15.78:0/2476483441 reap_dead start 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 shutdown_connections 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 >> 172.21.15.78:0/2476483441 conn(0x7fc3b40fc7d0 msgr2=0x7fc3b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 shutdown_connections 2024-09-16T01:09:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:42.810+0000 7fc3bbee0640 1 -- 172.21.15.78:0/2476483441 wait complete. 2024-09-16T01:09:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:43 smithi148 ceph-mon[35030]: pgmap v323: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-09-16T01:09:43.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:43 smithi078 ceph-mon[30937]: pgmap v323: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-09-16T01:09:43.496 INFO:teuthology.orchestra.run.smithi078.stdout:163208757307 2024-09-16T01:09:43.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-16T01:09:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:43 smithi193 ceph-mon[35701]: pgmap v323: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-09-16T01:09:44.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:44 smithi078 ceph-mon[30937]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-16T01:09:44.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 -- 172.21.15.78:0/2188353920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 msgr2=0x7fe9e81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:44.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 --2- 172.21.15.78:0/2188353920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e81021c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009900 tx=0x7fe9d802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:44.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 -- 172.21.15.78:0/2188353920 shutdown_connections 2024-09-16T01:09:44.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 --2- 172.21.15.78:0/2188353920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8106f10 0x7fe9e810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 --2- 172.21.15.78:0/2188353920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8102790 0x7fe9e81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 --2- 172.21.15.78:0/2188353920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e81021c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 -- 172.21.15.78:0/2188353920 >> 172.21.15.78:0/2188353920 conn(0x7fe9e80fd8c0 msgr2=0x7fe9e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 -- 172.21.15.78:0/2188353920 shutdown_connections 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.488+0000 7fe9ed648640 1 -- 172.21.15.78:0/2188353920 wait complete. 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 Processor -- start 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 -- start start 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8102790 0x7fe9e819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8106f10 0x7fe9e819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e8114320 con 0x7fe9e8101de0 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e8076340 con 0x7fe9e8102790 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9ed648640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e81144a0 con 0x7fe9e8106f10 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56186/0 (socket says 172.21.15.78:56186) 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9e6ffd640 1 -- 172.21.15.78:0/3240824943 learned_addr learned my addr 172.21.15.78:0/3240824943 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.489+0000 7fe9e77fe640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8106f10 0x7fe9e819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:44.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e67fc640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8102790 0x7fe9e819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 -- 172.21.15.78:0/3240824943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8106f10 msgr2=0x7fe9e819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8106f10 0x7fe9e819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 -- 172.21.15.78:0/3240824943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8102790 msgr2=0x7fe9e819acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8102790 0x7fe9e819acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 -- 172.21.15.78:0/3240824943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e819e6e0 con 0x7fe9e8101de0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9e6ffd640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8006010 tx=0x7fe9d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d803f030 con 0x7fe9e8101de0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8003ce0 con 0x7fe9e8101de0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e819e970 con 0x7fe9e8101de0 2024-09-16T01:09:44.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8035070 con 0x7fe9e8101de0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.490+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e81a7340 con 0x7fe9e8101de0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.491+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e81021c0 con 0x7fe9e8101de0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.492+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe9d8042030 con 0x7fe9e8101de0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.493+0000 7fe9cffff640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 0x7fe9d00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.493+0000 7fe9e67fc640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 0x7fe9d00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.493+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe9d80f0520 con 0x7fe9e8101de0 2024-09-16T01:09:44.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.493+0000 7fe9e67fc640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 0x7fe9d00a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc005fd0 tx=0x7fe9dc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:44.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.497+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe9d80bcde0 con 0x7fe9e8101de0 2024-09-16T01:09:44.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:44 smithi193 ceph-mon[35701]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-16T01:09:44.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:44 smithi148 ceph-mon[35030]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-16T01:09:44.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.669+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe9e810a0f0 con 0x7fe9e8101de0 2024-09-16T01:09:44.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.669+0000 7fe9cffff640 1 -- 172.21.15.78:0/3240824943 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe9d80bc6e0 con 0x7fe9e8101de0 2024-09-16T01:09:44.671 INFO:teuthology.orchestra.run.smithi078.stdout:107374182469 2024-09-16T01:09:44.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.671+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 msgr2=0x7fe9d00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:44.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 0x7fe9d00a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc005fd0 tx=0x7fe9dc005850 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 msgr2=0x7fe9e819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8006010 tx=0x7fe9d80042f0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 shutdown_connections 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe9d00a32d0 0x7fe9d00a5790 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9e8106f10 0x7fe9e819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9e8102790 0x7fe9e819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 --2- 172.21.15.78:0/3240824943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e8101de0 0x7fe9e819a790 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 >> 172.21.15.78:0/3240824943 conn(0x7fe9e80fd8c0 msgr2=0x7fe9e80fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 shutdown_connections 2024-09-16T01:09:44.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:44.672+0000 7fe9ed648640 1 -- 172.21.15.78:0/3240824943 wait complete. 2024-09-16T01:09:44.974 INFO:teuthology.orchestra.run.smithi078.stdout:137438953534 2024-09-16T01:09:44.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-16T01:09:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 -- 172.21.15.78:0/1942883051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102570 msgr2=0x7f01441029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 --2- 172.21.15.78:0/1942883051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102570 0x7f01441029b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0138009900 tx=0x7f013802a480 comp rx=0 tx=0).stop 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 -- 172.21.15.78:0/1942883051 shutdown_connections 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 --2- 172.21.15.78:0/1942883051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0144102ef0 0x7f014410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 --2- 172.21.15.78:0/1942883051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102570 0x7f01441029b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 --2- 172.21.15.78:0/1942883051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0144100e20 0x7f0144101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 -- 172.21.15.78:0/1942883051 >> 172.21.15.78:0/1942883051 conn(0x7f01440fc7d0 msgr2=0x7f01440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 -- 172.21.15.78:0/1942883051 shutdown_connections 2024-09-16T01:09:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.187+0000 7f014c2dd640 1 -- 172.21.15.78:0/1942883051 wait complete. 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 Processor -- start 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 -- start start 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0144100e20 0x7f01441985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0144102570 0x7f0144198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 0x7f014419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0144110f80 con 0x7f0144102ef0 2024-09-16T01:09:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0144110e00 con 0x7f0144102570 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014c2dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0144111100 con 0x7f0144100e20 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014a87a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 0x7f014419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014a87a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 0x7f014419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56204/0 (socket says 172.21.15.78:56204) 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014a079640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0144100e20 0x7f01441985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f014a87a640 1 -- 172.21.15.78:0/2634776742 learned_addr learned my addr 172.21.15.78:0/2634776742 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.188+0000 7f0149878640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0144102570 0x7f0144198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 -- 172.21.15.78:0/2634776742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0144100e20 msgr2=0x7f01441985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0144100e20 0x7f01441985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 -- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0144102570 msgr2=0x7f0144198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0144102570 0x7f0144198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014419d560 con 0x7f0144102ef0 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014a87a640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 0x7f014419ce80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c970 tx=0x7f013c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c007bb0 con 0x7f0144102ef0 2024-09-16T01:09:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014419d850 con 0x7f0144102ef0 2024-09-16T01:09:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f013c007d50 con 0x7f0144102ef0 2024-09-16T01:09:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c00f450 con 0x7f0144102ef0 2024-09-16T01:09:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.189+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01441a5130 con 0x7f0144102ef0 2024-09-16T01:09:45.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.190+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0144062980 con 0x7f0144102ef0 2024-09-16T01:09:45.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.191+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f013c004f60 con 0x7f0144102ef0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f01437fe640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f012c0a32d0 0x7f012c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f014a079640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f012c0a32d0 0x7f012c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f013c0cf9f0 con 0x7f0144102ef0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f01437fe640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] conn(0x7f012c0ae4a0 0x7f012c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f013c004490 con 0x7f012c0ae4a0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f013c0d42c0 con 0x7f0144102ef0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.192+0000 7f014a079640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f012c0a32d0 0x7f012c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0134005fd0 tx=0x7f0134009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.193+0000 7f0149878640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] conn(0x7f012c0ae4a0 0x7f012c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.193+0000 7f0149878640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] conn(0x7f012c0ae4a0 0x7f012c0b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.194+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== osd.10 v2:172.21.15.148:6824/85363667 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f013c004490 con 0x7f012c0ae4a0 2024-09-16T01:09:45.212 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:45 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3240824943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T01:09:45.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.214+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 --> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0144062e00 con 0x7f012c0ae4a0 2024-09-16T01:09:45.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.215+0000 7f01437fe640 1 -- 172.21.15.78:0/2634776742 <== osd.10 v2:172.21.15.148:6824/85363667 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0144062e00 con 0x7f012c0ae4a0 2024-09-16T01:09:45.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] conn(0x7f012c0ae4a0 msgr2=0x7f012c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] conn(0x7f012c0ae4a0 0x7f012c0b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f012c0a32d0 msgr2=0x7f012c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f012c0a32d0 0x7f012c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0134005fd0 tx=0x7f0134009680 comp rx=0 tx=0).stop 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 msgr2=0x7f014419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 --2- 172.21.15.78:0/2634776742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0144102ef0 0x7f014419ce80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c970 tx=0x7f013c00ce30 comp rx=0 tx=0).stop 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014a87a640 1 -- 172.21.15.78:0/2634776742 reap_dead start 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 shutdown_connections 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 >> 172.21.15.78:0/2634776742 conn(0x7f01440fc7d0 msgr2=0x7f014406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 shutdown_connections 2024-09-16T01:09:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.216+0000 7f014c2dd640 1 -- 172.21.15.78:0/2634776742 wait complete. 2024-09-16T01:09:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:45 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3240824943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T01:09:45.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:45 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3240824943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-16T01:09:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.639+0000 7f18939c0640 1 -- 172.21.15.78:0/1704046205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102570 msgr2=0x7f188c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.639+0000 7f18939c0640 1 --2- 172.21.15.78:0/1704046205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102570 0x7f188c1029b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1880009900 tx=0x7f188002a480 comp rx=0 tx=0).stop 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 -- 172.21.15.78:0/1704046205 shutdown_connections 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 --2- 172.21.15.78:0/1704046205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c102ef0 0x7f188c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 --2- 172.21.15.78:0/1704046205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102570 0x7f188c1029b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 --2- 172.21.15.78:0/1704046205 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f188c100e20 0x7f188c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 -- 172.21.15.78:0/1704046205 >> 172.21.15.78:0/1704046205 conn(0x7f188c0fc7d0 msgr2=0x7f188c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 -- 172.21.15.78:0/1704046205 shutdown_connections 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 -- 172.21.15.78:0/1704046205 wait complete. 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 Processor -- start 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 -- start start 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.640+0000 7f18939c0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c100e20 0x7f188c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f188c102570 0x7f188c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 0x7f188c1a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f188c110f80 con 0x7f188c102ef0 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f188c110e00 con 0x7f188c102570 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f188c111100 con 0x7f188c100e20 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 0x7f188c1a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 0x7f188c1a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56220/0 (socket says 172.21.15.78:56220) 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 -- 172.21.15.78:0/3215712195 learned_addr learned my addr 172.21.15.78:0/3215712195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f189175c640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c100e20 0x7f188c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1890f5b640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f188c102570 0x7f188c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 -- 172.21.15.78:0/3215712195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c100e20 msgr2=0x7f188c199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c100e20 0x7f188c199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 -- 172.21.15.78:0/3215712195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f188c102570 msgr2=0x7f188c199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f188c102570 0x7f188c199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c1a4910 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f1891f5d640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 0x7f188c1a43d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f188400c970 tx=0x7f188400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1884007bb0 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188c1a4ae0 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1884007d50 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188400f450 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.641+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188c1a4ff0 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.642+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f188c062980 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.643+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1884004ee0 con 0x7f188c102ef0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f188a7fc640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18740a32d0 0x7f18740a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f189175c640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18740a32d0 0x7f18740a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f18840cfe50 con 0x7f188c102ef0 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f188a7fc640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] conn(0x7f18740ae4a0 0x7f18740b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1884012ec0 con 0x7f18740ae4a0 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f18840d02a0 con 0x7f188c102ef0 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f1890f5b640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] conn(0x7f18740ae4a0 0x7f18740b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.644+0000 7f189175c640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18740a32d0 0x7f18740a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f187c005fd0 tx=0x7f187c009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.645+0000 7f1890f5b640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] conn(0x7f18740ae4a0 0x7f18740b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.645+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== osd.6 v2:172.21.15.78:6818/3932044692 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1884012ec0 con 0x7f18740ae4a0 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.658+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 --> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f188c062e00 con 0x7f18740ae4a0 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f188a7fc640 1 -- 172.21.15.78:0/3215712195 <== osd.6 v2:172.21.15.78:6818/3932044692 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f188c062e00 con 0x7f18740ae4a0 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] conn(0x7f18740ae4a0 msgr2=0x7f18740b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] conn(0x7f18740ae4a0 0x7f18740b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18740a32d0 msgr2=0x7f18740a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f18740a32d0 0x7f18740a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f187c005fd0 tx=0x7f187c009680 comp rx=0 tx=0).stop 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 msgr2=0x7f188c1a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 --2- 172.21.15.78:0/3215712195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f188c102ef0 0x7f188c1a43d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f188400c970 tx=0x7f188400ce30 comp rx=0 tx=0).stop 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f1891f5d640 1 -- 172.21.15.78:0/3215712195 reap_dead start 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 shutdown_connections 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 >> 172.21.15.78:0/3215712195 conn(0x7f188c0fc7d0 msgr2=0x7f188c106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 shutdown_connections 2024-09-16T01:09:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:45.659+0000 7f18939c0640 1 -- 172.21.15.78:0/3215712195 wait complete. 2024-09-16T01:09:46.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:46 smithi078 ceph-mon[30937]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-09-16T01:09:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:46 smithi193 ceph-mon[35701]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-09-16T01:09:46.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:46 smithi148 ceph-mon[35030]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-09-16T01:09:46.951 INFO:teuthology.orchestra.run.smithi078.stdout:300647710754 2024-09-16T01:09:46.951 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-16T01:09:47.522 INFO:teuthology.orchestra.run.smithi078.stdout:77309411401 2024-09-16T01:09:47.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-16T01:09:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:193273528373 2024-09-16T01:09:47.745 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-16T01:09:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:48.270 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182469 for osd.3 2024-09-16T01:09:48.270 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 -- 172.21.15.78:0/530961814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664106f10 msgr2=0x7fd66410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 --2- 172.21.15.78:0/530961814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664106f10 0x7fd66410ea90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009ca0 tx=0x7fd65c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 -- 172.21.15.78:0/530961814 shutdown_connections 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 --2- 172.21.15.78:0/530961814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664106f10 0x7fd66410ea90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 --2- 172.21.15.78:0/530961814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 0x7fd6641067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 --2- 172.21.15.78:0/530961814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664101de0 0x7fd6641021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 -- 172.21.15.78:0/530961814 >> 172.21.15.78:0/530961814 conn(0x7fd6640fd8c0 msgr2=0x7fd6640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 -- 172.21.15.78:0/530961814 shutdown_connections 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.197+0000 7fd6694ca640 1 -- 172.21.15.78:0/530961814 wait complete. 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 Processor -- start 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 -- start start 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 0x7fd664074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 0x7fd664074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111f70 con 0x7fd664101de0 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111df0 con 0x7fd664106f10 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6694ca640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641120f0 con 0x7fd664102790 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56230/0 (socket says 172.21.15.78:56230) 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 -- 172.21.15.78:0/3737225864 learned_addr learned my addr 172.21.15.78:0/3737225864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd65bfff640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 0x7fd664074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6637fe640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 0x7fd664074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 -- 172.21.15.78:0/3737225864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 msgr2=0x7fd664074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 0x7fd664074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 -- 172.21.15.78:0/3737225864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 msgr2=0x7fd664074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 0x7fd664074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd662ffd640 1 -- 172.21.15.78:0/3737225864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6641a68a0 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.198+0000 7fd6637fe640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 0x7fd664074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd662ffd640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c920 tx=0x7fd65400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654007c70 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654007e10 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6641a6b90 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654018890 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.199+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641a70d0 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.200+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6641021c0 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.201+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd654018a70 con 0x7fd664101de0 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.201+0000 7fd660ff9640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 0x7fd6500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.202+0000 7fd65bfff640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 0x7fd6500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.202+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd6540d0000 con 0x7fd664101de0 2024-09-16T01:09:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.202+0000 7fd65bfff640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 0x7fd6500a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd64c005fd0 tx=0x7fd64c00b510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.206+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd654014030 con 0x7fd664101de0 2024-09-16T01:09:49.361 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:49 smithi078 ceph-mon[30937]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-16T01:09:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.377+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd664075a20 con 0x7fd664101de0 2024-09-16T01:09:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.377+0000 7fd660ff9640 1 -- 172.21.15.78:0/3737225864 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd65400de20 con 0x7fd664101de0 2024-09-16T01:09:49.379 INFO:teuthology.orchestra.run.smithi078.stdout:249108103212 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 msgr2=0x7fd6500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 0x7fd6500a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd64c005fd0 tx=0x7fd64c00b510 comp rx=0 tx=0).stop 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 msgr2=0x7fd6640744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c920 tx=0x7fd65400cde0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 shutdown_connections 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd6500a32d0 0x7fd6500a5790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd664106f10 0x7fd664074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd664102790 0x7fd664074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 --2- 172.21.15.78:0/3737225864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd664101de0 0x7fd6640744f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 >> 172.21.15.78:0/3737225864 conn(0x7fd6640fd8c0 msgr2=0x7fd6640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 shutdown_connections 2024-09-16T01:09:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.379+0000 7fd6694ca640 1 -- 172.21.15.78:0/3737225864 wait complete. 2024-09-16T01:09:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:49 smithi148 ceph-mon[35030]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-16T01:09:49.520 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:49 smithi193 ceph-mon[35701]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3112446908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 msgr2=0x7f8bd41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3112446908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd41029b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009900 tx=0x7f8bc002a480 comp rx=0 tx=0).stop 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3112446908 shutdown_connections 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3112446908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3112446908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd41029b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3112446908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3112446908 >> 172.21.15.78:0/3112446908 conn(0x7f8bd40fc7d0 msgr2=0x7f8bd40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3112446908 shutdown_connections 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3112446908 wait complete. 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 Processor -- start 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.607+0000 7f8bdb1cc640 1 -- start start 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4110f80 con 0x7f8bd4102570 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4110e00 con 0x7f8bd4102ef0 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bdb1cc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd4111100 con 0x7f8bd4100e20 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56262/0 (socket says 172.21.15.78:56262) 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 -- 172.21.15.78:0/3103194814 learned_addr learned my addr 172.21.15.78:0/3103194814 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd8f68640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd9769640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 -- 172.21.15.78:0/3103194814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 msgr2=0x7f8bd40fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd40fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 -- 172.21.15.78:0/3103194814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 msgr2=0x7f8bd40fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd40fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 -- 172.21.15.78:0/3103194814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd41a6a70 con 0x7f8bd4102570 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd8f68640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd9769640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.608+0000 7f8bd3fff640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0006010 tx=0x7f8bc0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.609+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc003f070 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.609+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd41a6d00 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.609+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc0031c30 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.609+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0035030 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.609+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd41a71d0 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.610+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd4101200 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.611+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8bc0042070 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.612+0000 7f8bd1ffb640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 0x7f8bc40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.612+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8bc00f0540 con 0x7f8bd4102570 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.612+0000 7f8bd8f68640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 0x7f8bc40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.612+0000 7f8bd8f68640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 0x7f8bc40a5810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009c20 tx=0x7f8bc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.616+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8bc00bcda0 con 0x7f8bd4102570 2024-09-16T01:09:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/1164229244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 msgr2=0x7f0b681021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/1164229244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b681021c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009ca0 tx=0x7f0b5802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/1164229244 shutdown_connections 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/1164229244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 0x7f0b6810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/1164229244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 0x7f0b681067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/1164229244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b681021c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/1164229244 >> 172.21.15.78:0/1164229244 conn(0x7f0b680fd8c0 msgr2=0x7f0b680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/1164229244 shutdown_connections 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.441+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/1164229244 wait complete. 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 Processor -- start 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 -- start start 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 0x7f0b6819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68111f70 con 0x7f0b68101de0 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68111df0 con 0x7f0b68102790 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6f5d1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0b681120f0 con 0x7f0b68106f10 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6d36d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6d36d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56244/0 (socket says 172.21.15.78:56244) 2024-09-16T01:09:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.442+0000 7f0b6d36d640 1 -- 172.21.15.78:0/951389902 learned_addr learned my addr 172.21.15.78:0/951389902 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6db6e640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6cb6c640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 0x7f0b6819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stdout:55834574928 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 -- 172.21.15.78:0/951389902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 msgr2=0x7f0b6819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 -- 172.21.15.78:0/951389902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 msgr2=0x7f0b6819abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 0x7f0b6819abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 -- 172.21.15.78:0/951389902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6819f640 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6d36d640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b5802ade0 tx=0x7f0b580319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5802fa20 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b5802fbc0 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6819f8d0 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b580467b0 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.443+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b681a7210 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.444+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b681021c0 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.445+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0b58046950 con 0x7f0b68101de0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.446+0000 7f0b667fc640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 0x7f0b500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.446+0000 7f0b6cb6c640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 0x7f0b500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.446+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b580f1140 con 0x7f0b68101de0 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.446+0000 7f0b6cb6c640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 0x7f0b500a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0b680ff7d0 tx=0x7f0b5c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.450+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0b580bdb40 con 0x7f0b68101de0 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.624+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0b681a73e0 con 0x7f0b68101de0 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.625+0000 7f0b667fc640 1 -- 172.21.15.78:0/951389902 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b580bd440 con 0x7f0b68101de0 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 msgr2=0x7f0b500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 0x7f0b500a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0b680ff7d0 tx=0x7f0b5c00b390 comp rx=0 tx=0).stop 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 msgr2=0x7f0b6819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0b5802ade0 tx=0x7f0b580319f0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 shutdown_connections 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0b500a32d0 0x7f0b500a5790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b68102790 0x7f0b6819abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 --2- 172.21.15.78:0/951389902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b68101de0 0x7f0b6819a690 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 >> 172.21.15.78:0/951389902 conn(0x7f0b680fd8c0 msgr2=0x7f0b680fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 shutdown_connections 2024-09-16T01:09:49.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.627+0000 7f0b6f5d1640 1 -- 172.21.15.78:0/951389902 wait complete. 2024-09-16T01:09:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.815+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8bd410a0a0 con 0x7f8bd4102570 2024-09-16T01:09:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.816+0000 7f8bd1ffb640 1 -- 172.21.15.78:0/3103194814 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8bc00bc6a0 con 0x7f8bd4102570 2024-09-16T01:09:49.818 INFO:teuthology.orchestra.run.smithi078.stdout:163208757308 2024-09-16T01:09:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 msgr2=0x7f8bc40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 0x7f8bc40a5810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009c20 tx=0x7f8bc8005f90 comp rx=0 tx=0).stop 2024-09-16T01:09:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 msgr2=0x7f8bd40fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0006010 tx=0x7f8bc0003c10 comp rx=0 tx=0).stop 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 shutdown_connections 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f8bc40a3350 0x7f8bc40a5810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd4102ef0 0x7f8bd40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd4102570 0x7f8bd40fd1c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 --2- 172.21.15.78:0/3103194814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd4100e20 0x7f8bd40fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 >> 172.21.15.78:0/3103194814 conn(0x7f8bd40fc7d0 msgr2=0x7f8bd406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 shutdown_connections 2024-09-16T01:09:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:49.818+0000 7f8bdb1cc640 1 -- 172.21.15.78:0/3103194814 wait complete. 2024-09-16T01:09:50.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3737225864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T01:09:50.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/951389902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T01:09:50.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3103194814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T01:09:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3737225864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T01:09:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/951389902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T01:09:50.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3103194814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T01:09:50.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3737225864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-16T01:09:50.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/951389902' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-16T01:09:50.468 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3103194814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-16T01:09:50.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103209 got 249108103212 for osd.8 2024-09-16T01:09:50.922 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:51.245 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:51 smithi078 ceph-mon[30937]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:09:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:51 smithi148 ceph-mon[35030]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:09:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:51 smithi193 ceph-mon[35701]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:09:51.573 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 -- 172.21.15.78:0/199678357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014102d40 msgr2=0x7f0014103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 --2- 172.21.15.78:0/199678357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014102d40 0x7f0014103120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f000c009900 tx=0x7f000c02a480 comp rx=0 tx=0).stop 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 -- 172.21.15.78:0/199678357 shutdown_connections 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 --2- 172.21.15.78:0/199678357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f001410c170 0x7f001410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 --2- 172.21.15.78:0/199678357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0014103660 0x7f001410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 --2- 172.21.15.78:0/199678357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014102d40 0x7f0014103120 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 -- 172.21.15.78:0/199678357 >> 172.21.15.78:0/199678357 conn(0x7f00140fc7a0 msgr2=0x7f00140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 -- 172.21.15.78:0/199678357 shutdown_connections 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.859+0000 7f001bd23640 1 -- 172.21.15.78:0/199678357 wait complete. 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 Processor -- start 2024-09-16T01:09:51.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 -- start start 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0014102d40 0x7f0014198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f001410c170 0x7f001419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0014111fb0 con 0x7f0014103660 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0014111e30 con 0x7f0014102d40 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f001bd23640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0014112130 con 0x7f001410c170 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f00192be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f00192be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56294/0 (socket says 172.21.15.78:56294) 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.860+0000 7f00192be640 1 -- 172.21.15.78:0/3265795522 learned_addr learned my addr 172.21.15.78:0/3265795522 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f0019abf640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0014102d40 0x7f0014198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f001a2c0640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f001410c170 0x7f001419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:51.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 -- 172.21.15.78:0/3265795522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f001410c170 msgr2=0x7f001419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f001410c170 0x7f001419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 -- 172.21.15.78:0/3265795522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0014102d40 msgr2=0x7f0014198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0014102d40 0x7f0014198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 -- 172.21.15.78:0/3265795522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001419d440 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f00192be640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f000800c940 tx=0x7f000800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008004220 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00080043c0 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001419d730 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008004a00 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.861+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00141a5010 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.862+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0014103120 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.863+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f00080079b0 con 0x7f0014103660 2024-09-16T01:09:51.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.864+0000 7f0012ffd640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 0x7efffc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:51.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.864+0000 7f0019abf640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 0x7efffc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:51.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.864+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f00080cff00 con 0x7f0014103660 2024-09-16T01:09:51.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.864+0000 7f0019abf640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 0x7efffc0a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f000c0096c0 tx=0x7f000c031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:51.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:51.867+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0008014030 con 0x7f0014103660 2024-09-16T01:09:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.038+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f00141997f0 con 0x7f0014103660 2024-09-16T01:09:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.039+0000 7f0012ffd640 1 -- 172.21.15.78:0/3265795522 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f00141997f0 con 0x7f0014103660 2024-09-16T01:09:52.041 INFO:teuthology.orchestra.run.smithi078.stdout:330712481822 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 msgr2=0x7efffc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 0x7efffc0a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f000c0096c0 tx=0x7f000c031490 comp rx=0 tx=0).stop 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 msgr2=0x7f00141989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f000800c940 tx=0x7f000800ce00 comp rx=0 tx=0).stop 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 shutdown_connections 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7efffc0a32d0 0x7efffc0a5790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f001410c170 0x7f001419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0014103660 0x7f00141989d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 --2- 172.21.15.78:0/3265795522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0014102d40 0x7f0014198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 >> 172.21.15.78:0/3265795522 conn(0x7f00140fc7a0 msgr2=0x7f00140fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 shutdown_connections 2024-09-16T01:09:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.042+0000 7f001bd23640 1 -- 172.21.15.78:0/3265795522 wait complete. 2024-09-16T01:09:52.112 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:52 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3265795522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T01:09:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3936383979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8106a80 msgr2=0x7fb3f8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3936383979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8106a80 0x7fb3f8106e60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009930 tx=0x7fb3e802a420 comp rx=0 tx=0).stop 2024-09-16T01:09:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3936383979 shutdown_connections 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3936383979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8101420 0x7fb3f810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3936383979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8100a80 0x7fb3f8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3936383979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8106a80 0x7fb3f8106e60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3936383979 >> 172.21.15.78:0/3936383979 conn(0x7fb3f80fc7a0 msgr2=0x7fb3f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3936383979 shutdown_connections 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.102+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3936383979 wait complete. 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.103+0000 7fb3fe1e0640 1 Processor -- start 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.103+0000 7fb3fe1e0640 1 -- start start 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 0x7fb3f81974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 0x7fb3f819bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:52.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f8110f20 con 0x7fb3f8101420 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f8110da0 con 0x7fb3f8106a80 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f81110a0 con 0x7fb3f8100a80 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56324/0 (socket says 172.21.15.78:56324) 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3582857516 learned_addr learned my addr 172.21.15.78:0/3582857516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f77fe640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 0x7fb3f81974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f7fff640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 0x7fb3f819bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3582857516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 msgr2=0x7fb3f81974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 0x7fb3f81974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3582857516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 msgr2=0x7fb3f819bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 0x7fb3f819bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3582857516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f819c4a0 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f7fff640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 0x7fb3f819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f77fe640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 0x7fb3f81974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f6ffd640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b4a0 tx=0x7fb3ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec014070 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ec010af0 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec017d10 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f819c790 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.104+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f81a4070 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.105+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f8100f60 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.107+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb3ec017070 con 0x7fb3f8101420 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.107+0000 7fb3f4ff9640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 0x7fb3e00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.107+0000 7fb3f77fe640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 0x7fb3e00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:52.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.107+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb3ec0cf500 con 0x7fb3f8101420 2024-09-16T01:09:52.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.108+0000 7fb3f77fe640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 0x7fb3e00a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8000c00 tx=0x7fb3e8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:52.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.110+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb3ec09be70 con 0x7fb3f8101420 2024-09-16T01:09:52.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.279+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fb3f8198850 con 0x7fb3f8101420 2024-09-16T01:09:52.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.280+0000 7fb3f4ff9640 1 -- 172.21.15.78:0/3582857516 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb3ec09b770 con 0x7fb3f8101420 2024-09-16T01:09:52.282 INFO:teuthology.orchestra.run.smithi078.stdout:300647710755 2024-09-16T01:09:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 msgr2=0x7fb3e00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 0x7fb3e00a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8000c00 tx=0x7fb3e8005a20 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 msgr2=0x7fb3f8197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00b4a0 tx=0x7fb3ec00b960 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 shutdown_connections 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb3e00a32d0 0x7fb3e00a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3f8106a80 0x7fb3f819bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f8101420 0x7fb3f8197a30 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 --2- 172.21.15.78:0/3582857516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3f8100a80 0x7fb3f81974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 >> 172.21.15.78:0/3582857516 conn(0x7fb3f80fc7a0 msgr2=0x7fb3f80fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 shutdown_connections 2024-09-16T01:09:52.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:52.282+0000 7fb3fe1e0640 1 -- 172.21.15.78:0/3582857516 wait complete. 2024-09-16T01:09:52.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:52 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3265795522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T01:09:52.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:52 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3265795522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-16T01:09:53.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:53 smithi078 ceph-mon[30937]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:09:53.224 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:53 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3582857516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T01:09:53.258 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757307 got 163208757308 for osd.5 2024-09-16T01:09:53.258 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:53 smithi148 ceph-mon[35030]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:09:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:53 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3582857516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T01:09:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:53 smithi193 ceph-mon[35701]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:09:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:53 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3582857516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-16T01:09:53.820 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574928 for osd.1 2024-09-16T01:09:53.821 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:54.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 -- 172.21.15.78:0/790647901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8106f10 msgr2=0x7f07d810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:54.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 --2- 172.21.15.78:0/790647901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8106f10 0x7f07d810ea90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009900 tx=0x7f07c002a410 comp rx=0 tx=0).stop 2024-09-16T01:09:54.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 -- 172.21.15.78:0/790647901 shutdown_connections 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 --2- 172.21.15.78:0/790647901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8106f10 0x7f07d810ea90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 --2- 172.21.15.78:0/790647901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8102790 0x7f07d81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 --2- 172.21.15.78:0/790647901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 -- 172.21.15.78:0/790647901 >> 172.21.15.78:0/790647901 conn(0x7f07d80fd8c0 msgr2=0x7f07d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 -- 172.21.15.78:0/790647901 shutdown_connections 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.007+0000 7f07df220640 1 -- 172.21.15.78:0/790647901 wait complete. 2024-09-16T01:09:54.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 Processor -- start 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 -- start start 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8106f10 0x7f07d819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f07d8111f70 con 0x7f07d8102790 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f07d8111df0 con 0x7f07d8106f10 2024-09-16T01:09:54.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07df220640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07d81120f0 con 0x7f07d8101de0 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56344/0 (socket says 172.21.15.78:56344) 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 -- 172.21.15.78:0/857827855 learned_addr learned my addr 172.21.15.78:0/857827855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07dd7bd640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8106f10 0x7f07d819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07dcfbc640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 -- 172.21.15.78:0/857827855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 msgr2=0x7f07d81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:54.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 -- 172.21.15.78:0/857827855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8106f10 msgr2=0x7f07d819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8106f10 0x7f07d819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07d7fff640 1 -- 172.21.15.78:0/857827855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d819d360 con 0x7f07d8102790 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.008+0000 7f07dcfbc640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07d7fff640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f07c800cce0 tx=0x7f07c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:54.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c800f650 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07c800fcd0 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8018c70 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d819d650 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.009+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d81a4f30 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.010+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d81021c0 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.011+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f07c800f7f0 con 0x7f07d8102790 2024-09-16T01:09:54.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.012+0000 7f07d5ffb640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 0x7f07c40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.012+0000 7f07dcfbc640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 0x7f07c40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.012+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f07c80d0270 con 0x7f07d8102790 2024-09-16T01:09:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.012+0000 7f07dcfbc640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 0x7f07c40a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07d00058c0 tx=0x7f07d0005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.016+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f07c8014030 con 0x7f07d8102790 2024-09-16T01:09:54.044 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.192+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f07d81a5190 con 0x7f07d8102790 2024-09-16T01:09:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.193+0000 7f07d5ffb640 1 -- 172.21.15.78:0/857827855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f07c809cad0 con 0x7f07d8102790 2024-09-16T01:09:54.195 INFO:teuthology.orchestra.run.smithi078.stdout:219043332147 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 msgr2=0x7f07c40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 0x7f07c40a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07d00058c0 tx=0x7f07d0005d30 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 msgr2=0x7f07d81988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f07c800cce0 tx=0x7f07c800a620 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 shutdown_connections 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f07c40a3350 0x7f07c40a5810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07d8106f10 0x7f07d819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07d8102790 0x7f07d81988f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 --2- 172.21.15.78:0/857827855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07d8101de0 0x7f07d81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 >> 172.21.15.78:0/857827855 conn(0x7f07d80fd8c0 msgr2=0x7f07d80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 shutdown_connections 2024-09-16T01:09:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:54.195+0000 7f07df220640 1 -- 172.21.15.78:0/857827855 wait complete. 2024-09-16T01:09:54.967 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481819 got 330712481822 for osd.11 2024-09-16T01:09:54.967 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:55.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:55 smithi078 ceph-mon[30937]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:09:55.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:55 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/857827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T01:09:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:55 smithi148 ceph-mon[35030]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:09:55.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:55 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/857827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T01:09:55.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:55 smithi193 ceph-mon[35701]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:09:55.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:55 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/857827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-16T01:09:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 -- 172.21.15.78:0/4264910666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0101de0 msgr2=0x7f30b01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 --2- 172.21.15.78:0/4264910666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0101de0 0x7f30b01021c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f30a0009900 tx=0x7f30a002a410 comp rx=0 tx=0).stop 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 -- 172.21.15.78:0/4264910666 shutdown_connections 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 --2- 172.21.15.78:0/4264910666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0106f10 0x7f30b010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 --2- 172.21.15.78:0/4264910666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0102790 0x7f30b01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 --2- 172.21.15.78:0/4264910666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0101de0 0x7f30b01021c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 -- 172.21.15.78:0/4264910666 >> 172.21.15.78:0/4264910666 conn(0x7f30b00fd8c0 msgr2=0x7f30b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 -- 172.21.15.78:0/4264910666 shutdown_connections 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.604+0000 7f30b49f7640 1 -- 172.21.15.78:0/4264910666 wait complete. 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 Processor -- start 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 -- start start 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0101de0 0x7f30b00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0102790 0x7f30b0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f30b0111f90 con 0x7f30b0106f10 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30b0111e10 con 0x7f30b0102790 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30b49f7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30b0112110 con 0x7f30b0101de0 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30aed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30aed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42664/0 (socket says 172.21.15.78:42664) 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30aed9d640 1 -- 172.21.15.78:0/3887458550 learned_addr learned my addr 172.21.15.78:0/3887458550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30ae59c640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0101de0 0x7f30b00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.605+0000 7f30add9b640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0102790 0x7f30b0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 -- 172.21.15.78:0/3887458550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0101de0 msgr2=0x7f30b00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0101de0 0x7f30b00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 -- 172.21.15.78:0/3887458550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0102790 msgr2=0x7f30b0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0102790 0x7f30b0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 -- 172.21.15.78:0/3887458550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b01a6980 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30aed9d640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f30a800b970 tx=0x7f30a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8019070 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30a800cb40 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b01a6c70 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8007650 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.606+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b01a7160 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.607+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b01021c0 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.608+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f30a801c070 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.609+0000 7f309f7fe640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 0x7f30980a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.609+0000 7f30ae59c640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 0x7f30980a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.609+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f30a80cf330 con 0x7f30b0106f10 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.609+0000 7f30ae59c640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 0x7f30980a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f30a0000c00 tx=0x7f30a00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:55.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.613+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f30a809b5f0 con 0x7f30b0106f10 2024-09-16T01:09:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.791+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f30b0075a20 con 0x7f30b0106f10 2024-09-16T01:09:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.792+0000 7f309f7fe640 1 -- 172.21.15.78:0/3887458550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30a809f8f0 con 0x7f30b0106f10 2024-09-16T01:09:55.793 INFO:teuthology.orchestra.run.smithi078.stdout:77309411403 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.793+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 msgr2=0x7f30980a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.793+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 0x7f30980a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f30a0000c00 tx=0x7f30a00058d0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.793+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 msgr2=0x7f30b0074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.793+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f30a800b970 tx=0x7f30a800be30 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 shutdown_connections 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f30980a32d0 0x7f30980a5790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30b0106f10 0x7f30b0074f70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30b0102790 0x7f30b0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 --2- 172.21.15.78:0/3887458550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f30b0101de0 0x7f30b00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 >> 172.21.15.78:0/3887458550 conn(0x7f30b00fd8c0 msgr2=0x7f30b00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 shutdown_connections 2024-09-16T01:09:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.794+0000 7f30b49f7640 1 -- 172.21.15.78:0/3887458550 wait complete. 2024-09-16T01:09:55.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 -- 172.21.15.78:0/1210071091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034101de0 msgr2=0x7fe0341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 --2- 172.21.15.78:0/1210071091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034101de0 0x7fe0341021c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe028009900 tx=0x7fe02802a410 comp rx=0 tx=0).stop 2024-09-16T01:09:55.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 -- 172.21.15.78:0/1210071091 shutdown_connections 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 --2- 172.21.15.78:0/1210071091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034106f10 0x7fe03410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 --2- 172.21.15.78:0/1210071091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034102790 0x7fe0341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 --2- 172.21.15.78:0/1210071091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034101de0 0x7fe0341021c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 -- 172.21.15.78:0/1210071091 >> 172.21.15.78:0/1210071091 conn(0x7fe0340fd880 msgr2=0x7fe0340ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 -- 172.21.15.78:0/1210071091 shutdown_connections 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.832+0000 7fe03c05c640 1 -- 172.21.15.78:0/1210071091 wait complete. 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 Processor -- start 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 -- start start 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 0x7fe0341983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034102790 0x7fe0341988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe034111f70 con 0x7fe034106f10 2024-09-16T01:09:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe034111df0 con 0x7fe034101de0 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03c05c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0341120f0 con 0x7fe034102790 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42680/0 (socket says 172.21.15.78:42680) 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 -- 172.21.15.78:0/2753700734 learned_addr learned my addr 172.21.15.78:0/2753700734 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe0395f7640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034102790 0x7fe0341988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe039df8640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 0x7fe0341983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 -- 172.21.15.78:0/2753700734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034102790 msgr2=0x7fe0341988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034102790 0x7fe0341988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 -- 172.21.15.78:0/2753700734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 msgr2=0x7fe0341983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 0x7fe0341983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe03a5f9640 1 -- 172.21.15.78:0/2753700734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03419d360 con 0x7fe034106f10 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.833+0000 7fe039df8640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 0x7fe0341983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe03a5f9640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00eaa0 tx=0x7fe02c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c004d20 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03419d650 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe02c005020 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c0116d0 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.834+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0341a4f30 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.835+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0341021c0 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.836+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe02c0051d0 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.837+0000 7fe026ffd640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 0x7fe0200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.837+0000 7fe039df8640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 0x7fe0200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.837+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe02c0d0340 con 0x7fe034106f10 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.837+0000 7fe039df8640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 0x7fe0200a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe028000c00 tx=0x7fe028032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:55.840+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe02c016030 con 0x7fe034106f10 2024-09-16T01:09:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.009+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe034109b60 con 0x7fe034106f10 2024-09-16T01:09:56.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.009+0000 7fe026ffd640 1 -- 172.21.15.78:0/2753700734 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe02c09cba0 con 0x7fe034106f10 2024-09-16T01:09:56.011 INFO:teuthology.orchestra.run.smithi078.stdout:279172874281 2024-09-16T01:09:56.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 msgr2=0x7fe0200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 0x7fe0200a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe028000c00 tx=0x7fe028032c70 comp rx=0 tx=0).stop 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 msgr2=0x7fe03419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00eaa0 tx=0x7fe02c00eea0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 shutdown_connections 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe0200a32d0 0x7fe0200a5790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe034106f10 0x7fe03419cc80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe034102790 0x7fe0341988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 --2- 172.21.15.78:0/2753700734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe034101de0 0x7fe0341983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.011+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 >> 172.21.15.78:0/2753700734 conn(0x7fe0340fd880 msgr2=0x7fe0340feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:56.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.012+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 shutdown_connections 2024-09-16T01:09:56.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.012+0000 7fe03c05c640 1 -- 172.21.15.78:0/2753700734 wait complete. 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- 172.21.15.78:0/1314806599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 msgr2=0x7fa7441029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 --2- 172.21.15.78:0/1314806599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa7441029b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa734009900 tx=0x7fa73402a480 comp rx=0 tx=0).stop 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- 172.21.15.78:0/1314806599 shutdown_connections 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 --2- 172.21.15.78:0/1314806599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 0x7fa74410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 --2- 172.21.15.78:0/1314806599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa7441029b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 --2- 172.21.15.78:0/1314806599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 0x7fa744101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- 172.21.15.78:0/1314806599 >> 172.21.15.78:0/1314806599 conn(0x7fa7440fc7d0 msgr2=0x7fa7440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- 172.21.15.78:0/1314806599 shutdown_connections 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- 172.21.15.78:0/1314806599 wait complete. 2024-09-16T01:09:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 Processor -- start 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.096+0000 7fa74baad640 1 -- start start 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 0x7fa7441993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 0x7fa7441a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa744110f80 con 0x7fa744102570 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa744110e00 con 0x7fa744102ef0 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa744111100 con 0x7fa744100e20 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42690/0 (socket says 172.21.15.78:42690) 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 -- 172.21.15.78:0/752349882 learned_addr learned my addr 172.21.15.78:0/752349882 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74a04a640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 0x7fa7441a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749849640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 0x7fa7441993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 -- 172.21.15.78:0/752349882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 msgr2=0x7fa7441993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 0x7fa7441993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 -- 172.21.15.78:0/752349882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 msgr2=0x7fa7441a4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 0x7fa7441a4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 -- 172.21.15.78:0/752349882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7441a49a0 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa749048640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73403f070 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa734031c30 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7441a4b70 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734035030 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.097+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7441a4fc0 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.099+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa744101200 con 0x7fa744102570 2024-09-16T01:09:56.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.099+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa734042070 con 0x7fa744102570 2024-09-16T01:09:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.100+0000 7fa73affd640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 0x7fa7300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.100+0000 7fa749849640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 0x7fa7300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.100+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7340f0660 con 0x7fa744102570 2024-09-16T01:09:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.101+0000 7fa749849640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 0x7fa7300a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa740005fd0 tx=0x7fa740005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.104+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa734031de0 con 0x7fa744102570 2024-09-16T01:09:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.275+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa744109090 con 0x7fa744102570 2024-09-16T01:09:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.275+0000 7fa73affd640 1 -- 172.21.15.78:0/752349882 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa7340bcec0 con 0x7fa744102570 2024-09-16T01:09:56.277 INFO:teuthology.orchestra.run.smithi078.stdout:34359738454 2024-09-16T01:09:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 msgr2=0x7fa7300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 0x7fa7300a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa740005fd0 tx=0x7fa740005850 comp rx=0 tx=0).stop 2024-09-16T01:09:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 msgr2=0x7fa744199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734003c10 comp rx=0 tx=0).stop 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 shutdown_connections 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fa7300a32d0 0x7fa7300a5790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa744102ef0 0x7fa7441a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744102570 0x7fa744199920 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 --2- 172.21.15.78:0/752349882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744100e20 0x7fa7441993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 >> 172.21.15.78:0/752349882 conn(0x7fa7440fc7d0 msgr2=0x7fa7440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 shutdown_connections 2024-09-16T01:09:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:56.277+0000 7fa74baad640 1 -- 172.21.15.78:0/752349882 wait complete. 2024-09-16T01:09:56.369 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:56 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3887458550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T01:09:56.369 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:56 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2753700734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T01:09:56.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:56 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3887458550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T01:09:56.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:56 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2753700734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T01:09:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:56 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3887458550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-16T01:09:56.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:56 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2753700734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-16T01:09:57.102 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874278 got 279172874281 for osd.9 2024-09-16T01:09:57.103 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:57.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:57 smithi148 ceph-mon[35030]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:09:57.389 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:57 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/752349882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T01:09:57.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:57 smithi078 ceph-mon[30937]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:09:57.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:57 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/752349882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T01:09:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:57 smithi193 ceph-mon[35701]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:09:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:57 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/752349882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-16T01:09:57.663 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332144 got 219043332147 for osd.7 2024-09-16T01:09:57.663 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:58.317 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411403 for osd.2 2024-09-16T01:09:58.318 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:58.751 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738454 for osd.0 2024-09-16T01:09:58.751 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:59.037 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:09:59.069 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710754 got 300647710755 for osd.10 2024-09-16T01:09:59.069 DEBUG:teuthology.parallel:result is None 2024-09-16T01:09:59.345 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:09:59 smithi078 ceph-mon[30937]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-16T01:09:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:09:59 smithi148 ceph-mon[35030]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-16T01:09:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:09:59 smithi193 ceph-mon[35701]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 -- 172.21.15.78:0/210086655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 msgr2=0x7f5fe810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/210086655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810d950 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009cd0 tx=0x7f5fe002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 -- 172.21.15.78:0/210086655 shutdown_connections 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/210086655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810d950 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/210086655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8100a80 0x7f5fe8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/210086655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8106a80 0x7f5fe8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 -- 172.21.15.78:0/210086655 >> 172.21.15.78:0/210086655 conn(0x7f5fe80fc7a0 msgr2=0x7f5fe80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 -- 172.21.15.78:0/210086655 shutdown_connections 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.564+0000 7f5fefc1f640 1 -- 172.21.15.78:0/210086655 wait complete. 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 Processor -- start 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 -- start start 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 0x7f5fe810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8106a80 0x7f5fe810ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8111fa0 con 0x7f5fe8101420 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8111e20 con 0x7f5fe8100a80 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fefc1f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8112120 con 0x7f5fe8106a80 2024-09-16T01:09:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed1ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed1ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42714/0 (socket says 172.21.15.78:42714) 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed1ba640 1 -- 172.21.15.78:0/1803250380 learned_addr learned my addr 172.21.15.78:0/1803250380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed9bb640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 0x7f5fe810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fee1bc640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8106a80 0x7f5fe810ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed1ba640 1 -- 172.21.15.78:0/1803250380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8106a80 msgr2=0x7f5fe810ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.565+0000 7f5fed1ba640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8106a80 0x7f5fe810ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fed1ba640 1 -- 172.21.15.78:0/1803250380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 msgr2=0x7f5fe810e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fed1ba640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 0x7f5fe810e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fed1ba640 1 -- 172.21.15.78:0/1803250380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe81a6840 con 0x7f5fe8101420 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fed9bb640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 0x7f5fe810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fed1ba640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00e9f0 tx=0x7f5fdc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc00cd80 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fdc005500 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe81a6b30 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc010690 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.566+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe81a7020 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.567+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe8100ee0 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.568+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5fdc0046c0 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.569+0000 7f5fdaffd640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 0x7f5fd40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.569+0000 7f5fed9bb640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 0x7f5fd40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.569+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5fdc0d0340 con 0x7f5fe8101420 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.569+0000 7f5fed9bb640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 0x7f5fd40a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400baa0 tx=0x7f5fe4005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:09:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.573+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5fdc09cba0 con 0x7f5fe8101420 2024-09-16T01:09:59.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.745+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5fe810f910 con 0x7f5fe8101420 2024-09-16T01:09:59.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.745+0000 7f5fdaffd640 1 -- 172.21.15.78:0/1803250380 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5fdc09c4a0 con 0x7f5fe8101420 2024-09-16T01:09:59.746 INFO:teuthology.orchestra.run.smithi078.stdout:193273528376 2024-09-16T01:09:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 msgr2=0x7f5fd40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 0x7f5fd40a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5fe400baa0 tx=0x7f5fe4005d20 comp rx=0 tx=0).stop 2024-09-16T01:09:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 msgr2=0x7f5fe810e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:09:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00e9f0 tx=0x7f5fdc00edf0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 shutdown_connections 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f5fd40a32d0 0x7f5fd40a5790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fe8106a80 0x7f5fe810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fe8101420 0x7f5fe810e920 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 --2- 172.21.15.78:0/1803250380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5fe8100a80 0x7f5fe810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 >> 172.21.15.78:0/1803250380 conn(0x7f5fe80fc7a0 msgr2=0x7f5fe80fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 shutdown_connections 2024-09-16T01:09:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:09:59.747+0000 7f5fefc1f640 1 -- 172.21.15.78:0/1803250380 wait complete. 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.260+0000 7f06363e0640 1 -- 172.21.15.78:0/600393104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 msgr2=0x7f063010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.260+0000 7f06363e0640 1 --2- 172.21.15.78:0/600393104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063010da40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0628009ca0 tx=0x7f062802a950 comp rx=0 tx=0).stop 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 -- 172.21.15.78:0/600393104 shutdown_connections 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 --2- 172.21.15.78:0/600393104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063010da40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 --2- 172.21.15.78:0/600393104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630102570 0x7f06301029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 --2- 172.21.15.78:0/600393104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630100e20 0x7f0630101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 -- 172.21.15.78:0/600393104 >> 172.21.15.78:0/600393104 conn(0x7f06300fc7d0 msgr2=0x7f06300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 -- 172.21.15.78:0/600393104 shutdown_connections 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 -- 172.21.15.78:0/600393104 wait complete. 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 Processor -- start 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.261+0000 7f06363e0640 1 -- start start 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630100e20 0x7f063019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630102570 0x7f063019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0630110f80 con 0x7f0630102ef0 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0630110e00 con 0x7f0630102570 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0630111100 con 0x7f0630100e20 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42720/0 (socket says 172.21.15.78:42720) 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 -- 172.21.15.78:0/1977093453 learned_addr learned my addr 172.21.15.78:0/1977093453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f062ffff640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630100e20 0x7f063019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f062f7fe640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630102570 0x7f063019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 -- 172.21.15.78:0/1977093453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630100e20 msgr2=0x7f063019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630100e20 0x7f063019a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 -- 172.21.15.78:0/1977093453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630102570 msgr2=0x7f063019aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630102570 0x7f063019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 -- 172.21.15.78:0/1977093453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063019f4e0 con 0x7f0630102ef0 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f063497d640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0628006010 tx=0x7f062802f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:00.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628034040 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0628003e00 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063019f770 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628045480 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.262+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06301a70b0 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.263+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06301021b0 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.265+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0628037020 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.265+0000 7f062d7fa640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 0x7f06180a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.266+0000 7f062ffff640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 0x7f06180a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.266+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f06280f0f20 con 0x7f0630102ef0 2024-09-16T01:10:00.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.266+0000 7f062ffff640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 0x7f06180a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f062000ba40 tx=0x7f0620005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:00.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.270+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f06280bd890 con 0x7f0630102ef0 2024-09-16T01:10:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:00 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1803250380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T01:10:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:00 smithi193 ceph-mon[35701]: overall HEALTH_OK 2024-09-16T01:10:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:00 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1803250380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T01:10:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:00 smithi148 ceph-mon[35030]: overall HEALTH_OK 2024-09-16T01:10:00.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.442+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f06301a7280 con 0x7f0630102ef0 2024-09-16T01:10:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:00 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1803250380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-16T01:10:00.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:00 smithi078 ceph-mon[30937]: overall HEALTH_OK 2024-09-16T01:10:00.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.443+0000 7f062d7fa640 1 -- 172.21.15.78:0/1977093453 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f06280bd190 con 0x7f0630102ef0 2024-09-16T01:10:00.445 INFO:teuthology.orchestra.run.smithi078.stdout:137438953538 2024-09-16T01:10:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 msgr2=0x7f06180a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 0x7f06180a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f062000ba40 tx=0x7f0620005cc0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 msgr2=0x7f063019ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:00.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0628006010 tx=0x7f062802f6f0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 shutdown_connections 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f06180a32d0 0x7f06180a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0630102ef0 0x7f063019ee00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0630102570 0x7f063019aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 --2- 172.21.15.78:0/1977093453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0630100e20 0x7f063019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 >> 172.21.15.78:0/1977093453 conn(0x7f06300fc7d0 msgr2=0x7f063006a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 shutdown_connections 2024-09-16T01:10:00.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:00.445+0000 7f06363e0640 1 -- 172.21.15.78:0/1977093453 wait complete. 2024-09-16T01:10:00.839 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528373 got 193273528376 for osd.6 2024-09-16T01:10:00.839 DEBUG:teuthology.parallel:result is None 2024-09-16T01:10:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:01 smithi148 ceph-mon[35030]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:10:01.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:01 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1977093453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T01:10:01.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:01 smithi078 ceph-mon[30937]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:10:01.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:01 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1977093453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T01:10:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:01 smithi193 ceph-mon[35701]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-16T01:10:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:01 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1977093453' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-16T01:10:01.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953534 got 137438953538 for osd.4 2024-09-16T01:10:01.547 DEBUG:teuthology.parallel:result is None 2024-09-16T01:10:01.548 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T01:10:01.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:10:01.798 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:02.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 -- 172.21.15.78:0/438406887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8102790 msgr2=0x7f78e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:02.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/438406887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8102790 0x7f78e81067e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009900 tx=0x7f78dc02a410 comp rx=0 tx=0).stop 2024-09-16T01:10:02.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 -- 172.21.15.78:0/438406887 shutdown_connections 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/438406887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8106f10 0x7f78e810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/438406887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8102790 0x7f78e81067e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/438406887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78e8101de0 0x7f78e81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 -- 172.21.15.78:0/438406887 >> 172.21.15.78:0/438406887 conn(0x7f78e80fd8c0 msgr2=0x7f78e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 -- 172.21.15.78:0/438406887 shutdown_connections 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.282+0000 7f78ee3a7640 1 -- 172.21.15.78:0/438406887 wait complete. 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.283+0000 7f78ee3a7640 1 Processor -- start 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.283+0000 7f78ee3a7640 1 -- start start 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78e8102790 0x7f78e81989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8106f10 0x7f78e819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f78e8111f90 con 0x7f78e8101de0 2024-09-16T01:10:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78e8111e10 con 0x7f78e8106f10 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78e8112110 con 0x7f78e8102790 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42730/0 (socket says 172.21.15.78:42730) 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 -- 172.21.15.78:0/1268556659 learned_addr learned my addr 172.21.15.78:0/1268556659 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 -- 172.21.15.78:0/1268556659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78e8102790 msgr2=0x7f78e81989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78e8102790 0x7f78e81989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 -- 172.21.15.78:0/1268556659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8106f10 msgr2=0x7f78e819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8106f10 0x7f78e819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 -- 172.21.15.78:0/1268556659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e819d440 con 0x7f78e8101de0 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e7fff640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b4a0 tx=0x7f78d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8014070 con 0x7f78e8101de0 2024-09-16T01:10:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d8010af0 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e819d730 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8017d10 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.284+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e81a5010 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.285+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e81021c0 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.286+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f78d8017070 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.287+0000 7f78e57fa640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 0x7f78d00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.287+0000 7f78e77fe640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 0x7f78d00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.287+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f78d80cf100 con 0x7f78e8101de0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.288+0000 7f78e77fe640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 0x7f78d00a5810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f78dc02a930 tx=0x7f78dc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.291+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f78d809ba70 con 0x7f78e8101de0 2024-09-16T01:10:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.455+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78e802d830 con 0x7f78d00a3350 2024-09-16T01:10:02.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.463+0000 7f78e57fa640 1 -- 172.21.15.78:0/1268556659 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340621 (secure 0 0 0) 0x7f78e802d830 con 0x7f78d00a3350 2024-09-16T01:10:02.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:02.470 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:10:02.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 msgr2=0x7f78d00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:02.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 0x7f78d00a5810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f78dc02a930 tx=0x7f78dc000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 msgr2=0x7f78e8198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:02.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f78d800b4a0 tx=0x7f78d800b960 comp rx=0 tx=0).stop 2024-09-16T01:10:02.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 shutdown_connections 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f78d00a3350 0x7f78d00a5810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8106f10 0x7f78e819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78e8102790 0x7f78e81989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 --2- 172.21.15.78:0/1268556659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e8101de0 0x7f78e8198490 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.466+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 >> 172.21.15.78:0/1268556659 conn(0x7f78e80fd8c0 msgr2=0x7f78e80fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.467+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 shutdown_connections 2024-09-16T01:10:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:02.467+0000 7f78ee3a7640 1 -- 172.21.15.78:0/1268556659 wait complete. 2024-09-16T01:10:03.305 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-16T01:10:02.014861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656681,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":483,"num_read_kb":359,"num_write":353,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374000,"kb_used_data":12200,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567824,"statfs":{"total":1151940427776,"available":1151557451776,"internally_reserved":0,"allocated":12492800,"data_stored":8048633,"data_compressed":389616,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003023"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113170+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:09:39.113170+0000","last_peered":"2024-09-16T01:09:39.113170+0000","last_clean":"2024-09-16T01:09:39.113170+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:09:39.113170+0000","last_undegraded":"2024-09-16T01:09:39.113170+0000","last_fullsized":"2024-09-16T01:09:39.113170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147863+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:09:39.147863+0000","last_peered":"2024-09-16T01:09:39.147863+0000","last_clean":"2024-09-16T01:09:39.147863+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:09:39.147863+0000","last_undegraded":"2024-09-16T01:09:39.147863+0000","last_fullsized":"2024-09-16T01:09:39.147863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110930+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:09:39.110930+0000","last_peered":"2024-09-16T01:09:39.110930+0000","last_clean":"2024-09-16T01:09:39.110930+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:09:39.110930+0000","last_undegraded":"2024-09-16T01:09:39.110930+0000","last_fullsized":"2024-09-16T01:09:39.110930+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148084+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:09:39.148084+0000","last_peered":"2024-09-16T01:09:39.148084+0000","last_clean":"2024-09-16T01:09:39.148084+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:09:39.148084+0000","last_undegraded":"2024-09-16T01:09:39.148084+0000","last_fullsized":"2024-09-16T01:09:39.148084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148378+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:09:39.148378+0000","last_peered":"2024-09-16T01:09:39.148378+0000","last_clean":"2024-09-16T01:09:39.148378+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:09:39.148378+0000","last_undegraded":"2024-09-16T01:09:39.148378+0000","last_fullsized":"2024-09-16T01:09:39.148378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111007+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:09:39.111007+0000","last_peered":"2024-09-16T01:09:39.111007+0000","last_clean":"2024-09-16T01:09:39.111007+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:09:39.111007+0000","last_undegraded":"2024-09-16T01:09:39.111007+0000","last_fullsized":"2024-09-16T01:09:39.111007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112077+0000","last_change":"2024-09-16T01:08:26.711638+0000","last_active":"2024-09-16T01:09:39.112077+0000","last_peered":"2024-09-16T01:09:39.112077+0000","last_clean":"2024-09-16T01:09:39.112077+0000","last_became_active":"2024-09-16T01:08:26.711038+0000","last_became_peered":"2024-09-16T01:08:26.711038+0000","last_unstale":"2024-09-16T01:09:39.112077+0000","last_undegraded":"2024-09-16T01:09:39.112077+0000","last_fullsized":"2024-09-16T01:09:39.112077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:12.619264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109060+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:09:39.109060+0000","last_peered":"2024-09-16T01:09:39.109060+0000","last_clean":"2024-09-16T01:09:39.109060+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:09:39.109060+0000","last_undegraded":"2024-09-16T01:09:39.109060+0000","last_fullsized":"2024-09-16T01:09:39.109060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111228+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:09:39.111228+0000","last_peered":"2024-09-16T01:09:39.111228+0000","last_clean":"2024-09-16T01:09:39.111228+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:09:39.111228+0000","last_undegraded":"2024-09-16T01:09:39.111228+0000","last_fullsized":"2024-09-16T01:09:39.111228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112881+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:09:39.112881+0000","last_peered":"2024-09-16T01:09:39.112881+0000","last_clean":"2024-09-16T01:09:39.112881+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:09:39.112881+0000","last_undegraded":"2024-09-16T01:09:39.112881+0000","last_fullsized":"2024-09-16T01:09:39.112881+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114931+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:09:39.114931+0000","last_peered":"2024-09-16T01:09:39.114931+0000","last_clean":"2024-09-16T01:09:39.114931+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:09:39.114931+0000","last_undegraded":"2024-09-16T01:09:39.114931+0000","last_fullsized":"2024-09-16T01:09:39.114931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111080+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:09:39.111080+0000","last_peered":"2024-09-16T01:09:39.111080+0000","last_clean":"2024-09-16T01:09:39.111080+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:09:39.111080+0000","last_undegraded":"2024-09-16T01:09:39.111080+0000","last_fullsized":"2024-09-16T01:09:39.111080+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146944+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:09:39.146944+0000","last_peered":"2024-09-16T01:09:39.146944+0000","last_clean":"2024-09-16T01:09:39.146944+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:09:39.146944+0000","last_undegraded":"2024-09-16T01:09:39.146944+0000","last_fullsized":"2024-09-16T01:09:39.146944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110936+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:09:39.110936+0000","last_peered":"2024-09-16T01:09:39.110936+0000","last_clean":"2024-09-16T01:09:39.110936+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:09:39.110936+0000","last_undegraded":"2024-09-16T01:09:39.110936+0000","last_fullsized":"2024-09-16T01:09:39.110936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111508+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:09:39.111508+0000","last_peered":"2024-09-16T01:09:39.111508+0000","last_clean":"2024-09-16T01:09:39.111508+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:09:39.111508+0000","last_undegraded":"2024-09-16T01:09:39.111508+0000","last_fullsized":"2024-09-16T01:09:39.111508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147427+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:09:39.147427+0000","last_peered":"2024-09-16T01:09:39.147427+0000","last_clean":"2024-09-16T01:09:39.147427+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:09:39.147427+0000","last_undegraded":"2024-09-16T01:09:39.147427+0000","last_fullsized":"2024-09-16T01:09:39.147427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111811+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:09:39.111811+0000","last_peered":"2024-09-16T01:09:39.111811+0000","last_clean":"2024-09-16T01:09:39.111811+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:09:39.111811+0000","last_undegraded":"2024-09-16T01:09:39.111811+0000","last_fullsized":"2024-09-16T01:09:39.111811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111496+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:09:39.111496+0000","last_peered":"2024-09-16T01:09:39.111496+0000","last_clean":"2024-09-16T01:09:39.111496+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:09:39.111496+0000","last_undegraded":"2024-09-16T01:09:39.111496+0000","last_fullsized":"2024-09-16T01:09:39.111496+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113684+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:09:39.113684+0000","last_peered":"2024-09-16T01:09:39.113684+0000","last_clean":"2024-09-16T01:09:39.113684+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:09:39.113684+0000","last_undegraded":"2024-09-16T01:09:39.113684+0000","last_fullsized":"2024-09-16T01:09:39.113684+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147516+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:09:39.147516+0000","last_peered":"2024-09-16T01:09:39.147516+0000","last_clean":"2024-09-16T01:09:39.147516+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:09:39.147516+0000","last_undegraded":"2024-09-16T01:09:39.147516+0000","last_fullsized":"2024-09-16T01:09:39.147516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112990+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:09:39.112990+0000","last_peered":"2024-09-16T01:09:39.112990+0000","last_clean":"2024-09-16T01:09:39.112990+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:09:39.112990+0000","last_undegraded":"2024-09-16T01:09:39.112990+0000","last_fullsized":"2024-09-16T01:09:39.112990+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112392+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:09:39.112392+0000","last_peered":"2024-09-16T01:09:39.112392+0000","last_clean":"2024-09-16T01:09:39.112392+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:09:39.112392+0000","last_undegraded":"2024-09-16T01:09:39.112392+0000","last_fullsized":"2024-09-16T01:09:39.112392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109096+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:09:39.109096+0000","last_peered":"2024-09-16T01:09:39.109096+0000","last_clean":"2024-09-16T01:09:39.109096+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:09:39.109096+0000","last_undegraded":"2024-09-16T01:09:39.109096+0000","last_fullsized":"2024-09-16T01:09:39.109096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111693+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:09:39.111693+0000","last_peered":"2024-09-16T01:09:39.111693+0000","last_clean":"2024-09-16T01:09:39.111693+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:09:39.111693+0000","last_undegraded":"2024-09-16T01:09:39.111693+0000","last_fullsized":"2024-09-16T01:09:39.111693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148073+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:09:39.148073+0000","last_peered":"2024-09-16T01:09:39.148073+0000","last_clean":"2024-09-16T01:09:39.148073+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:09:39.148073+0000","last_undegraded":"2024-09-16T01:09:39.148073+0000","last_fullsized":"2024-09-16T01:09:39.148073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112533+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:09:39.112533+0000","last_peered":"2024-09-16T01:09:39.112533+0000","last_clean":"2024-09-16T01:09:39.112533+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:09:39.112533+0000","last_undegraded":"2024-09-16T01:09:39.112533+0000","last_fullsized":"2024-09-16T01:09:39.112533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111140+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:09:39.111140+0000","last_peered":"2024-09-16T01:09:39.111140+0000","last_clean":"2024-09-16T01:09:39.111140+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:09:39.111140+0000","last_undegraded":"2024-09-16T01:09:39.111140+0000","last_fullsized":"2024-09-16T01:09:39.111140+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148167+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:09:39.148167+0000","last_peered":"2024-09-16T01:09:39.148167+0000","last_clean":"2024-09-16T01:09:39.148167+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:09:39.148167+0000","last_undegraded":"2024-09-16T01:09:39.148167+0000","last_fullsized":"2024-09-16T01:09:39.148167+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112996+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:09:39.112996+0000","last_peered":"2024-09-16T01:09:39.112996+0000","last_clean":"2024-09-16T01:09:39.112996+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:09:39.112996+0000","last_undegraded":"2024-09-16T01:09:39.112996+0000","last_fullsized":"2024-09-16T01:09:39.112996+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147127+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:09:39.147127+0000","last_peered":"2024-09-16T01:09:39.147127+0000","last_clean":"2024-09-16T01:09:39.147127+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:09:39.147127+0000","last_undegraded":"2024-09-16T01:09:39.147127+0000","last_fullsized":"2024-09-16T01:09:39.147127+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167691+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:09:39.167691+0000","last_peered":"2024-09-16T01:09:39.167691+0000","last_clean":"2024-09-16T01:09:39.167691+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:09:39.167691+0000","last_undegraded":"2024-09-16T01:09:39.167691+0000","last_fullsized":"2024-09-16T01:09:39.167691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112238+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:09:39.112238+0000","last_peered":"2024-09-16T01:09:39.112238+0000","last_clean":"2024-09-16T01:09:39.112238+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:09:39.112238+0000","last_undegraded":"2024-09-16T01:09:39.112238+0000","last_fullsized":"2024-09-16T01:09:39.112238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114197+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:09:39.114197+0000","last_peered":"2024-09-16T01:09:39.114197+0000","last_clean":"2024-09-16T01:09:39.114197+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:09:39.114197+0000","last_undegraded":"2024-09-16T01:09:39.114197+0000","last_fullsized":"2024-09-16T01:09:39.114197+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114058+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:09:39.114058+0000","last_peered":"2024-09-16T01:09:39.114058+0000","last_clean":"2024-09-16T01:09:39.114058+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:09:39.114058+0000","last_undegraded":"2024-09-16T01:09:39.114058+0000","last_fullsized":"2024-09-16T01:09:39.114058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112855+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:09:39.112855+0000","last_peered":"2024-09-16T01:09:39.112855+0000","last_clean":"2024-09-16T01:09:39.112855+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:09:39.112855+0000","last_undegraded":"2024-09-16T01:09:39.112855+0000","last_fullsized":"2024-09-16T01:09:39.112855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.117797+0000","last_change":"2024-09-16T01:09:39.117797+0000","last_active":"2024-09-16T01:09:39.117797+0000","last_peered":"2024-09-16T01:09:39.117797+0000","last_clean":"2024-09-16T01:09:39.117797+0000","last_became_active":"2024-09-16T01:09:39.117345+0000","last_became_peered":"2024-09-16T01:09:39.117345+0000","last_unstale":"2024-09-16T01:09:39.117797+0000","last_undegraded":"2024-09-16T01:09:39.117797+0000","last_fullsized":"2024-09-16T01:09:39.117797+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:21.410065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148653+0000","last_change":"2024-09-16T01:09:39.148653+0000","last_active":"2024-09-16T01:09:39.148653+0000","last_peered":"2024-09-16T01:09:39.148653+0000","last_clean":"2024-09-16T01:09:39.148653+0000","last_became_active":"2024-09-16T01:09:39.148312+0000","last_became_peered":"2024-09-16T01:09:39.148312+0000","last_unstale":"2024-09-16T01:09:39.148653+0000","last_undegraded":"2024-09-16T01:09:39.148653+0000","last_fullsized":"2024-09-16T01:09:39.148653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:17.031741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147056+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:09:39.147056+0000","last_peered":"2024-09-16T01:09:39.147056+0000","last_clean":"2024-09-16T01:09:39.147056+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:09:39.147056+0000","last_undegraded":"2024-09-16T01:09:39.147056+0000","last_fullsized":"2024-09-16T01:09:39.147056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167484+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:09:39.167484+0000","last_peered":"2024-09-16T01:09:39.167484+0000","last_clean":"2024-09-16T01:09:39.167484+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:09:39.167484+0000","last_undegraded":"2024-09-16T01:09:39.167484+0000","last_fullsized":"2024-09-16T01:09:39.167484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114488+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:09:39.114488+0000","last_peered":"2024-09-16T01:09:39.114488+0000","last_clean":"2024-09-16T01:09:39.114488+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:09:39.114488+0000","last_undegraded":"2024-09-16T01:09:39.114488+0000","last_fullsized":"2024-09-16T01:09:39.114488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:56.190241+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:09:56.190241+0000","last_peered":"2024-09-16T01:09:56.190241+0000","last_clean":"2024-09-16T01:09:56.190241+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:09:56.190241+0000","last_undegraded":"2024-09-16T01:09:56.190241+0000","last_fullsized":"2024-09-16T01:09:56.190241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167631+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:09:39.167631+0000","last_peered":"2024-09-16T01:09:39.167631+0000","last_clean":"2024-09-16T01:09:39.167631+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:09:39.167631+0000","last_undegraded":"2024-09-16T01:09:39.167631+0000","last_fullsized":"2024-09-16T01:09:39.167631+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167093+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:09:39.167093+0000","last_peered":"2024-09-16T01:09:39.167093+0000","last_clean":"2024-09-16T01:09:39.167093+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:09:39.167093+0000","last_undegraded":"2024-09-16T01:09:39.167093+0000","last_fullsized":"2024-09-16T01:09:39.167093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146926+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:09:39.146926+0000","last_peered":"2024-09-16T01:09:39.146926+0000","last_clean":"2024-09-16T01:09:39.146926+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:09:39.146926+0000","last_undegraded":"2024-09-16T01:09:39.146926+0000","last_fullsized":"2024-09-16T01:09:39.146926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167998+0000","last_change":"2024-09-16T01:09:39.167998+0000","last_active":"2024-09-16T01:09:39.167998+0000","last_peered":"2024-09-16T01:09:39.167998+0000","last_clean":"2024-09-16T01:09:39.167998+0000","last_became_active":"2024-09-16T01:09:39.167533+0000","last_became_peered":"2024-09-16T01:09:39.167533+0000","last_unstale":"2024-09-16T01:09:39.167998+0000","last_undegraded":"2024-09-16T01:09:39.167998+0000","last_fullsized":"2024-09-16T01:09:39.167998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:01.974490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147780+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:09:39.147780+0000","last_peered":"2024-09-16T01:09:39.147780+0000","last_clean":"2024-09-16T01:09:39.147780+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:09:39.147780+0000","last_undegraded":"2024-09-16T01:09:39.147780+0000","last_fullsized":"2024-09-16T01:09:39.147780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112127+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:09:39.112127+0000","last_peered":"2024-09-16T01:09:39.112127+0000","last_clean":"2024-09-16T01:09:39.112127+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:09:39.112127+0000","last_undegraded":"2024-09-16T01:09:39.112127+0000","last_fullsized":"2024-09-16T01:09:39.112127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147002+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:09:39.147002+0000","last_peered":"2024-09-16T01:09:39.147002+0000","last_clean":"2024-09-16T01:09:39.147002+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:09:39.147002+0000","last_undegraded":"2024-09-16T01:09:39.147002+0000","last_fullsized":"2024-09-16T01:09:39.147002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147152+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:09:39.147152+0000","last_peered":"2024-09-16T01:09:39.147152+0000","last_clean":"2024-09-16T01:09:39.147152+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:09:39.147152+0000","last_undegraded":"2024-09-16T01:09:39.147152+0000","last_fullsized":"2024-09-16T01:09:39.147152+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147316+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:09:39.147316+0000","last_peered":"2024-09-16T01:09:39.147316+0000","last_clean":"2024-09-16T01:09:39.147316+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:09:39.147316+0000","last_undegraded":"2024-09-16T01:09:39.147316+0000","last_fullsized":"2024-09-16T01:09:39.147316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112362+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:09:39.112362+0000","last_peered":"2024-09-16T01:09:39.112362+0000","last_clean":"2024-09-16T01:09:39.112362+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:09:39.112362+0000","last_undegraded":"2024-09-16T01:09:39.112362+0000","last_fullsized":"2024-09-16T01:09:39.112362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.108858+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:09:39.108858+0000","last_peered":"2024-09-16T01:09:39.108858+0000","last_clean":"2024-09-16T01:09:39.108858+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:09:39.108858+0000","last_undegraded":"2024-09-16T01:09:39.108858+0000","last_fullsized":"2024-09-16T01:09:39.108858+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147125+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:09:39.147125+0000","last_peered":"2024-09-16T01:09:39.147125+0000","last_clean":"2024-09-16T01:09:39.147125+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:09:39.147125+0000","last_undegraded":"2024-09-16T01:09:39.147125+0000","last_fullsized":"2024-09-16T01:09:39.147125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115189+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:09:39.115189+0000","last_peered":"2024-09-16T01:09:39.115189+0000","last_clean":"2024-09-16T01:09:39.115189+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:09:39.115189+0000","last_undegraded":"2024-09-16T01:09:39.115189+0000","last_fullsized":"2024-09-16T01:09:39.115189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147445+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:09:39.147445+0000","last_peered":"2024-09-16T01:09:39.147445+0000","last_clean":"2024-09-16T01:09:39.147445+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:09:39.147445+0000","last_undegraded":"2024-09-16T01:09:39.147445+0000","last_fullsized":"2024-09-16T01:09:39.147445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147319+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:09:39.147319+0000","last_peered":"2024-09-16T01:09:39.147319+0000","last_clean":"2024-09-16T01:09:39.147319+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:09:39.147319+0000","last_undegraded":"2024-09-16T01:09:39.147319+0000","last_fullsized":"2024-09-16T01:09:39.147319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167673+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:09:39.167673+0000","last_peered":"2024-09-16T01:09:39.167673+0000","last_clean":"2024-09-16T01:09:39.167673+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:09:39.167673+0000","last_undegraded":"2024-09-16T01:09:39.167673+0000","last_fullsized":"2024-09-16T01:09:39.167673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114298+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:09:39.114298+0000","last_peered":"2024-09-16T01:09:39.114298+0000","last_clean":"2024-09-16T01:09:39.114298+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:09:39.114298+0000","last_undegraded":"2024-09-16T01:09:39.114298+0000","last_fullsized":"2024-09-16T01:09:39.114298+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.119140+0000","last_change":"2024-09-16T01:09:39.119140+0000","last_active":"2024-09-16T01:09:39.119140+0000","last_peered":"2024-09-16T01:09:39.119140+0000","last_clean":"2024-09-16T01:09:39.119140+0000","last_became_active":"2024-09-16T01:09:39.118679+0000","last_became_peered":"2024-09-16T01:09:39.118679+0000","last_unstale":"2024-09-16T01:09:39.119140+0000","last_undegraded":"2024-09-16T01:09:39.119140+0000","last_fullsized":"2024-09-16T01:09:39.119140+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:14.355461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111730+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:09:39.111730+0000","last_peered":"2024-09-16T01:09:39.111730+0000","last_clean":"2024-09-16T01:09:39.111730+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:09:39.111730+0000","last_undegraded":"2024-09-16T01:09:39.111730+0000","last_fullsized":"2024-09-16T01:09:39.111730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115575+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:09:39.115575+0000","last_peered":"2024-09-16T01:09:39.115575+0000","last_clean":"2024-09-16T01:09:39.115575+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:09:39.115575+0000","last_undegraded":"2024-09-16T01:09:39.115575+0000","last_fullsized":"2024-09-16T01:09:39.115575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121178+0000","last_change":"2024-09-16T01:09:39.121178+0000","last_active":"2024-09-16T01:09:39.121178+0000","last_peered":"2024-09-16T01:09:39.121178+0000","last_clean":"2024-09-16T01:09:39.121178+0000","last_became_active":"2024-09-16T01:09:39.120781+0000","last_became_peered":"2024-09-16T01:09:39.120781+0000","last_unstale":"2024-09-16T01:09:39.121178+0000","last_undegraded":"2024-09-16T01:09:39.121178+0000","last_fullsized":"2024-09-16T01:09:39.121178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:43.670325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110770+0000","last_change":"2024-09-16T01:08:26.956123+0000","last_active":"2024-09-16T01:09:39.110770+0000","last_peered":"2024-09-16T01:09:39.110770+0000","last_clean":"2024-09-16T01:09:39.110770+0000","last_became_active":"2024-09-16T01:08:26.955746+0000","last_became_peered":"2024-09-16T01:08:26.955746+0000","last_unstale":"2024-09-16T01:09:39.110770+0000","last_undegraded":"2024-09-16T01:09:39.110770+0000","last_fullsized":"2024-09-16T01:09:39.110770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:40.575736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167656+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:09:39.167656+0000","last_peered":"2024-09-16T01:09:39.167656+0000","last_clean":"2024-09-16T01:09:39.167656+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:09:39.167656+0000","last_undegraded":"2024-09-16T01:09:39.167656+0000","last_fullsized":"2024-09-16T01:09:39.167656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146816+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:09:39.146816+0000","last_peered":"2024-09-16T01:09:39.146816+0000","last_clean":"2024-09-16T01:09:39.146816+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:09:39.146816+0000","last_undegraded":"2024-09-16T01:09:39.146816+0000","last_fullsized":"2024-09-16T01:09:39.146816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":96,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:59.195595+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:09:59.195595+0000","last_peered":"2024-09-16T01:09:59.195595+0000","last_clean":"2024-09-16T01:09:59.195595+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:09:59.195595+0000","last_undegraded":"2024-09-16T01:09:59.195595+0000","last_fullsized":"2024-09-16T01:09:59.195595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113069+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:09:39.113069+0000","last_peered":"2024-09-16T01:09:39.113069+0000","last_clean":"2024-09-16T01:09:39.113069+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:09:39.113069+0000","last_undegraded":"2024-09-16T01:09:39.113069+0000","last_fullsized":"2024-09-16T01:09:39.113069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147170+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:09:39.147170+0000","last_peered":"2024-09-16T01:09:39.147170+0000","last_clean":"2024-09-16T01:09:39.147170+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:09:39.147170+0000","last_undegraded":"2024-09-16T01:09:39.147170+0000","last_fullsized":"2024-09-16T01:09:39.147170+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115237+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:09:39.115237+0000","last_peered":"2024-09-16T01:09:39.115237+0000","last_clean":"2024-09-16T01:09:39.115237+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:09:39.115237+0000","last_undegraded":"2024-09-16T01:09:39.115237+0000","last_fullsized":"2024-09-16T01:09:39.115237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114958+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:09:39.114958+0000","last_peered":"2024-09-16T01:09:39.114958+0000","last_clean":"2024-09-16T01:09:39.114958+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:09:39.114958+0000","last_undegraded":"2024-09-16T01:09:39.114958+0000","last_fullsized":"2024-09-16T01:09:39.114958+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167295+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:09:39.167295+0000","last_peered":"2024-09-16T01:09:39.167295+0000","last_clean":"2024-09-16T01:09:39.167295+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:09:39.167295+0000","last_undegraded":"2024-09-16T01:09:39.167295+0000","last_fullsized":"2024-09-16T01:09:39.167295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111749+0000","last_change":"2024-09-16T01:08:26.710927+0000","last_active":"2024-09-16T01:09:39.111749+0000","last_peered":"2024-09-16T01:09:39.111749+0000","last_clean":"2024-09-16T01:09:39.111749+0000","last_became_active":"2024-09-16T01:08:26.709796+0000","last_became_peered":"2024-09-16T01:08:26.709796+0000","last_unstale":"2024-09-16T01:09:39.111749+0000","last_undegraded":"2024-09-16T01:09:39.111749+0000","last_fullsized":"2024-09-16T01:09:39.111749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:30.270120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114817+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:09:39.114817+0000","last_peered":"2024-09-16T01:09:39.114817+0000","last_clean":"2024-09-16T01:09:39.114817+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:09:39.114817+0000","last_undegraded":"2024-09-16T01:09:39.114817+0000","last_fullsized":"2024-09-16T01:09:39.114817+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111895+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:09:39.111895+0000","last_peered":"2024-09-16T01:09:39.111895+0000","last_clean":"2024-09-16T01:09:39.111895+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:09:39.111895+0000","last_undegraded":"2024-09-16T01:09:39.111895+0000","last_fullsized":"2024-09-16T01:09:39.111895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:57.192349+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:09:57.192349+0000","last_peered":"2024-09-16T01:09:57.192349+0000","last_clean":"2024-09-16T01:09:57.192349+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:09:57.192349+0000","last_undegraded":"2024-09-16T01:09:57.192349+0000","last_fullsized":"2024-09-16T01:09:57.192349+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62076,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":117,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121127+0000","last_change":"2024-09-16T01:09:39.121127+0000","last_active":"2024-09-16T01:09:39.121127+0000","last_peered":"2024-09-16T01:09:39.121127+0000","last_clean":"2024-09-16T01:09:39.121127+0000","last_became_active":"2024-09-16T01:09:39.120736+0000","last_became_peered":"2024-09-16T01:09:39.120736+0000","last_unstale":"2024-09-16T01:09:39.121127+0000","last_undegraded":"2024-09-16T01:09:39.121127+0000","last_fullsized":"2024-09-16T01:09:39.121127+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:28.281410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.168566+0000","last_change":"2024-09-16T01:09:39.168566+0000","last_active":"2024-09-16T01:09:39.168566+0000","last_peered":"2024-09-16T01:09:39.168566+0000","last_clean":"2024-09-16T01:09:39.168566+0000","last_became_active":"2024-09-16T01:09:39.168127+0000","last_became_peered":"2024-09-16T01:09:39.168127+0000","last_unstale":"2024-09-16T01:09:39.168566+0000","last_undegraded":"2024-09-16T01:09:39.168566+0000","last_fullsized":"2024-09-16T01:09:39.168566+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:52.054331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115057+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:09:39.115057+0000","last_peered":"2024-09-16T01:09:39.115057+0000","last_clean":"2024-09-16T01:09:39.115057+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:09:39.115057+0000","last_undegraded":"2024-09-16T01:09:39.115057+0000","last_fullsized":"2024-09-16T01:09:39.115057+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121335+0000","last_change":"2024-09-16T01:09:39.121335+0000","last_active":"2024-09-16T01:09:39.121335+0000","last_peered":"2024-09-16T01:09:39.121335+0000","last_clean":"2024-09-16T01:09:39.121335+0000","last_became_active":"2024-09-16T01:09:39.120738+0000","last_became_peered":"2024-09-16T01:09:39.120738+0000","last_unstale":"2024-09-16T01:09:39.121335+0000","last_undegraded":"2024-09-16T01:09:39.121335+0000","last_fullsized":"2024-09-16T01:09:39.121335+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:09.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148388+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:09:39.148388+0000","last_peered":"2024-09-16T01:09:39.148388+0000","last_clean":"2024-09-16T01:09:39.148388+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:09:39.148388+0000","last_undegraded":"2024-09-16T01:09:39.148388+0000","last_fullsized":"2024-09-16T01:09:39.148388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147219+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:09:39.147219+0000","last_peered":"2024-09-16T01:09:39.147219+0000","last_clean":"2024-09-16T01:09:39.147219+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:09:39.147219+0000","last_undegraded":"2024-09-16T01:09:39.147219+0000","last_fullsized":"2024-09-16T01:09:39.147219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112418+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:09:39.112418+0000","last_peered":"2024-09-16T01:09:39.112418+0000","last_clean":"2024-09-16T01:09:39.112418+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:09:39.112418+0000","last_undegraded":"2024-09-16T01:09:39.112418+0000","last_fullsized":"2024-09-16T01:09:39.112418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109375+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:09:39.109375+0000","last_peered":"2024-09-16T01:09:39.109375+0000","last_clean":"2024-09-16T01:09:39.109375+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:09:39.109375+0000","last_undegraded":"2024-09-16T01:09:39.109375+0000","last_fullsized":"2024-09-16T01:09:39.109375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:56.190882+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:09:56.190882+0000","last_peered":"2024-09-16T01:09:56.190882+0000","last_clean":"2024-09-16T01:09:56.190882+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:09:56.190882+0000","last_undegraded":"2024-09-16T01:09:56.190882+0000","last_fullsized":"2024-09-16T01:09:56.190882+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147270+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:09:39.147270+0000","last_peered":"2024-09-16T01:09:39.147270+0000","last_clean":"2024-09-16T01:09:39.147270+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:09:39.147270+0000","last_undegraded":"2024-09-16T01:09:39.147270+0000","last_fullsized":"2024-09-16T01:09:39.147270+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147905+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:09:39.147905+0000","last_peered":"2024-09-16T01:09:39.147905+0000","last_clean":"2024-09-16T01:09:39.147905+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:09:39.147905+0000","last_undegraded":"2024-09-16T01:09:39.147905+0000","last_fullsized":"2024-09-16T01:09:39.147905+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112495+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:09:39.112495+0000","last_peered":"2024-09-16T01:09:39.112495+0000","last_clean":"2024-09-16T01:09:39.112495+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:09:39.112495+0000","last_undegraded":"2024-09-16T01:09:39.112495+0000","last_fullsized":"2024-09-16T01:09:39.112495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114701+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:09:39.114701+0000","last_peered":"2024-09-16T01:09:39.114701+0000","last_clean":"2024-09-16T01:09:39.114701+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:09:39.114701+0000","last_undegraded":"2024-09-16T01:09:39.114701+0000","last_fullsized":"2024-09-16T01:09:39.114701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114341+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:09:39.114341+0000","last_peered":"2024-09-16T01:09:39.114341+0000","last_clean":"2024-09-16T01:09:39.114341+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:09:39.114341+0000","last_undegraded":"2024-09-16T01:09:39.114341+0000","last_fullsized":"2024-09-16T01:09:39.114341+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147328+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:09:39.147328+0000","last_peered":"2024-09-16T01:09:39.147328+0000","last_clean":"2024-09-16T01:09:39.147328+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:09:39.147328+0000","last_undegraded":"2024-09-16T01:09:39.147328+0000","last_fullsized":"2024-09-16T01:09:39.147328+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167078+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:09:39.167078+0000","last_peered":"2024-09-16T01:09:39.167078+0000","last_clean":"2024-09-16T01:09:39.167078+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:09:39.167078+0000","last_undegraded":"2024-09-16T01:09:39.167078+0000","last_fullsized":"2024-09-16T01:09:39.167078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114306+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:09:39.114306+0000","last_peered":"2024-09-16T01:09:39.114306+0000","last_clean":"2024-09-16T01:09:39.114306+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:09:39.114306+0000","last_undegraded":"2024-09-16T01:09:39.114306+0000","last_fullsized":"2024-09-16T01:09:39.114306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114765+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:09:39.114765+0000","last_peered":"2024-09-16T01:09:39.114765+0000","last_clean":"2024-09-16T01:09:39.114765+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:09:39.114765+0000","last_undegraded":"2024-09-16T01:09:39.114765+0000","last_fullsized":"2024-09-16T01:09:39.114765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111232+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:09:39.111232+0000","last_peered":"2024-09-16T01:09:39.111232+0000","last_clean":"2024-09-16T01:09:39.111232+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:09:39.111232+0000","last_undegraded":"2024-09-16T01:09:39.111232+0000","last_fullsized":"2024-09-16T01:09:39.111232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:10:00.197765+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:10:00.197765+0000","last_peered":"2024-09-16T01:10:00.197765+0000","last_clean":"2024-09-16T01:10:00.197765+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:10:00.197765+0000","last_undegraded":"2024-09-16T01:10:00.197765+0000","last_fullsized":"2024-09-16T01:10:00.197765+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112366+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:09:39.112366+0000","last_peered":"2024-09-16T01:09:39.112366+0000","last_clean":"2024-09-16T01:09:39.112366+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:09:39.112366+0000","last_undegraded":"2024-09-16T01:09:39.112366+0000","last_fullsized":"2024-09-16T01:09:39.112366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111619+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:09:39.111619+0000","last_peered":"2024-09-16T01:09:39.111619+0000","last_clean":"2024-09-16T01:09:39.111619+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:09:39.111619+0000","last_undegraded":"2024-09-16T01:09:39.111619+0000","last_fullsized":"2024-09-16T01:09:39.111619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.119041+0000","last_change":"2024-09-16T01:09:39.119041+0000","last_active":"2024-09-16T01:09:39.119041+0000","last_peered":"2024-09-16T01:09:39.119041+0000","last_clean":"2024-09-16T01:09:39.119041+0000","last_became_active":"2024-09-16T01:09:39.118531+0000","last_became_peered":"2024-09-16T01:09:39.118531+0000","last_unstale":"2024-09-16T01:09:39.119041+0000","last_undegraded":"2024-09-16T01:09:39.119041+0000","last_fullsized":"2024-09-16T01:09:39.119041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:32:18.412355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112004+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:09:39.112004+0000","last_peered":"2024-09-16T01:09:39.112004+0000","last_clean":"2024-09-16T01:09:39.112004+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:09:39.112004+0000","last_undegraded":"2024-09-16T01:09:39.112004+0000","last_fullsized":"2024-09-16T01:09:39.112004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111015+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:09:39.111015+0000","last_peered":"2024-09-16T01:09:39.111015+0000","last_clean":"2024-09-16T01:09:39.111015+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:09:39.111015+0000","last_undegraded":"2024-09-16T01:09:39.111015+0000","last_fullsized":"2024-09-16T01:09:39.111015+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147535+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:09:39.147535+0000","last_peered":"2024-09-16T01:09:39.147535+0000","last_clean":"2024-09-16T01:09:39.147535+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:09:39.147535+0000","last_undegraded":"2024-09-16T01:09:39.147535+0000","last_fullsized":"2024-09-16T01:09:39.147535+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167637+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:09:39.167637+0000","last_peered":"2024-09-16T01:09:39.167637+0000","last_clean":"2024-09-16T01:09:39.167637+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:09:39.167637+0000","last_undegraded":"2024-09-16T01:09:39.167637+0000","last_fullsized":"2024-09-16T01:09:39.167637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113887+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:09:39.113887+0000","last_peered":"2024-09-16T01:09:39.113887+0000","last_clean":"2024-09-16T01:09:39.113887+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:09:39.113887+0000","last_undegraded":"2024-09-16T01:09:39.113887+0000","last_fullsized":"2024-09-16T01:09:39.113887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112241+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:09:39.112241+0000","last_peered":"2024-09-16T01:09:39.112241+0000","last_clean":"2024-09-16T01:09:39.112241+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:09:39.112241+0000","last_undegraded":"2024-09-16T01:09:39.112241+0000","last_fullsized":"2024-09-16T01:09:39.112241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148075+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:09:39.148075+0000","last_peered":"2024-09-16T01:09:39.148075+0000","last_clean":"2024-09-16T01:09:39.148075+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:09:39.148075+0000","last_undegraded":"2024-09-16T01:09:39.148075+0000","last_fullsized":"2024-09-16T01:09:39.148075+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121080+0000","last_change":"2024-09-16T01:09:39.121080+0000","last_active":"2024-09-16T01:09:39.121080+0000","last_peered":"2024-09-16T01:09:39.121080+0000","last_clean":"2024-09-16T01:09:39.121080+0000","last_became_active":"2024-09-16T01:09:39.120707+0000","last_became_peered":"2024-09-16T01:09:39.120707+0000","last_unstale":"2024-09-16T01:09:39.121080+0000","last_undegraded":"2024-09-16T01:09:39.121080+0000","last_fullsized":"2024-09-16T01:09:39.121080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:08.521558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147425+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:09:39.147425+0000","last_peered":"2024-09-16T01:09:39.147425+0000","last_clean":"2024-09-16T01:09:39.147425+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:09:39.147425+0000","last_undegraded":"2024-09-16T01:09:39.147425+0000","last_fullsized":"2024-09-16T01:09:39.147425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109196+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:09:39.109196+0000","last_peered":"2024-09-16T01:09:39.109196+0000","last_clean":"2024-09-16T01:09:39.109196+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:09:39.109196+0000","last_undegraded":"2024-09-16T01:09:39.109196+0000","last_fullsized":"2024-09-16T01:09:39.109196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167487+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:09:39.167487+0000","last_peered":"2024-09-16T01:09:39.167487+0000","last_clean":"2024-09-16T01:09:39.167487+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:09:39.167487+0000","last_undegraded":"2024-09-16T01:09:39.167487+0000","last_fullsized":"2024-09-16T01:09:39.167487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115061+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:09:39.115061+0000","last_peered":"2024-09-16T01:09:39.115061+0000","last_clean":"2024-09-16T01:09:39.115061+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:09:39.115061+0000","last_undegraded":"2024-09-16T01:09:39.115061+0000","last_fullsized":"2024-09-16T01:09:39.115061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111783+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:09:39.111783+0000","last_peered":"2024-09-16T01:09:39.111783+0000","last_clean":"2024-09-16T01:09:39.111783+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:09:39.111783+0000","last_undegraded":"2024-09-16T01:09:39.111783+0000","last_fullsized":"2024-09-16T01:09:39.111783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112954+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:09:39.112954+0000","last_peered":"2024-09-16T01:09:39.112954+0000","last_clean":"2024-09-16T01:09:39.112954+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:09:39.112954+0000","last_undegraded":"2024-09-16T01:09:39.112954+0000","last_fullsized":"2024-09-16T01:09:39.112954+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114746+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:09:39.114746+0000","last_peered":"2024-09-16T01:09:39.114746+0000","last_clean":"2024-09-16T01:09:39.114746+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:09:39.114746+0000","last_undegraded":"2024-09-16T01:09:39.114746+0000","last_fullsized":"2024-09-16T01:09:39.114746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.108831+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:09:39.108831+0000","last_peered":"2024-09-16T01:09:39.108831+0000","last_clean":"2024-09-16T01:09:39.108831+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:09:39.108831+0000","last_undegraded":"2024-09-16T01:09:39.108831+0000","last_fullsized":"2024-09-16T01:09:39.108831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147754+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:09:39.147754+0000","last_peered":"2024-09-16T01:09:39.147754+0000","last_clean":"2024-09-16T01:09:39.147754+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:09:39.147754+0000","last_undegraded":"2024-09-16T01:09:39.147754+0000","last_fullsized":"2024-09-16T01:09:39.147754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147372+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:09:39.147372+0000","last_peered":"2024-09-16T01:09:39.147372+0000","last_clean":"2024-09-16T01:09:39.147372+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:09:39.147372+0000","last_undegraded":"2024-09-16T01:09:39.147372+0000","last_fullsized":"2024-09-16T01:09:39.147372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167450+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:09:39.167450+0000","last_peered":"2024-09-16T01:09:39.167450+0000","last_clean":"2024-09-16T01:09:39.167450+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:09:39.167450+0000","last_undegraded":"2024-09-16T01:09:39.167450+0000","last_fullsized":"2024-09-16T01:09:39.167450+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111871+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:09:39.111871+0000","last_peered":"2024-09-16T01:09:39.111871+0000","last_clean":"2024-09-16T01:09:39.111871+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:09:39.111871+0000","last_undegraded":"2024-09-16T01:09:39.111871+0000","last_fullsized":"2024-09-16T01:09:39.111871+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147313+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:09:39.147313+0000","last_peered":"2024-09-16T01:09:39.147313+0000","last_clean":"2024-09-16T01:09:39.147313+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:09:39.147313+0000","last_undegraded":"2024-09-16T01:09:39.147313+0000","last_fullsized":"2024-09-16T01:09:39.147313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114426+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:09:39.114426+0000","last_peered":"2024-09-16T01:09:39.114426+0000","last_clean":"2024-09-16T01:09:39.114426+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:09:39.114426+0000","last_undegraded":"2024-09-16T01:09:39.114426+0000","last_fullsized":"2024-09-16T01:09:39.114426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109572+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:09:39.109572+0000","last_peered":"2024-09-16T01:09:39.109572+0000","last_clean":"2024-09-16T01:09:39.109572+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:09:39.109572+0000","last_undegraded":"2024-09-16T01:09:39.109572+0000","last_fullsized":"2024-09-16T01:09:39.109572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147238+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:09:39.147238+0000","last_peered":"2024-09-16T01:09:39.147238+0000","last_clean":"2024-09-16T01:09:39.147238+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:09:39.147238+0000","last_undegraded":"2024-09-16T01:09:39.147238+0000","last_fullsized":"2024-09-16T01:09:39.147238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146902+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:09:39.146902+0000","last_peered":"2024-09-16T01:09:39.146902+0000","last_clean":"2024-09-16T01:09:39.146902+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:09:39.146902+0000","last_undegraded":"2024-09-16T01:09:39.146902+0000","last_fullsized":"2024-09-16T01:09:39.146902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114244+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:09:39.114244+0000","last_peered":"2024-09-16T01:09:39.114244+0000","last_clean":"2024-09-16T01:09:39.114244+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:09:39.114244+0000","last_undegraded":"2024-09-16T01:09:39.114244+0000","last_fullsized":"2024-09-16T01:09:39.114244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111065+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:09:39.111065+0000","last_peered":"2024-09-16T01:09:39.111065+0000","last_clean":"2024-09-16T01:09:39.111065+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:09:39.111065+0000","last_undegraded":"2024-09-16T01:09:39.111065+0000","last_fullsized":"2024-09-16T01:09:39.111065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111750+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:09:39.111750+0000","last_peered":"2024-09-16T01:09:39.111750+0000","last_clean":"2024-09-16T01:09:39.111750+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:09:39.111750+0000","last_undegraded":"2024-09-16T01:09:39.111750+0000","last_fullsized":"2024-09-16T01:09:39.111750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147807+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:09:39.147807+0000","last_peered":"2024-09-16T01:09:39.147807+0000","last_clean":"2024-09-16T01:09:39.147807+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:09:39.147807+0000","last_undegraded":"2024-09-16T01:09:39.147807+0000","last_fullsized":"2024-09-16T01:09:39.147807+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147988+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:09:39.147988+0000","last_peered":"2024-09-16T01:09:39.147988+0000","last_clean":"2024-09-16T01:09:39.147988+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:09:39.147988+0000","last_undegraded":"2024-09-16T01:09:39.147988+0000","last_fullsized":"2024-09-16T01:09:39.147988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114641+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:09:39.114641+0000","last_peered":"2024-09-16T01:09:39.114641+0000","last_clean":"2024-09-16T01:09:39.114641+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:09:39.114641+0000","last_undegraded":"2024-09-16T01:09:39.114641+0000","last_fullsized":"2024-09-16T01:09:39.114641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66079,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":347,"num_read_kb":240,"num_write":170,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282755,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481824,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":999424,"data_stored":508555,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710757,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1142784,"data_stored":1158292,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874282,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1019904,"data_stored":512887,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103215,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1015808,"data_stored":508711,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1064960,"data_stored":1104622,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528376,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":503967,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1073152,"data_stored":1099817,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":509010,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":504161,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":567303,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1060864,"data_stored":565953,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1019904,"data_stored":505355,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:10:03.314 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:10:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:03 smithi148 ceph-mon[35030]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:03 smithi078 ceph-mon[30937]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:03 smithi193 ceph-mon[35701]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:03.560 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 -- 172.21.15.78:0/4225680330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878106f10 msgr2=0x7f287810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 --2- 172.21.15.78:0/4225680330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878106f10 0x7f287810ea90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2868009900 tx=0x7f286802a410 comp rx=0 tx=0).stop 2024-09-16T01:10:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 -- 172.21.15.78:0/4225680330 shutdown_connections 2024-09-16T01:10:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 --2- 172.21.15.78:0/4225680330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878106f10 0x7f287810ea90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 --2- 172.21.15.78:0/4225680330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878102790 0x7f28781067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 --2- 172.21.15.78:0/4225680330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878101de0 0x7f28781021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 -- 172.21.15.78:0/4225680330 >> 172.21.15.78:0/4225680330 conn(0x7f28780fd8c0 msgr2=0x7f28780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 -- 172.21.15.78:0/4225680330 shutdown_connections 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 -- 172.21.15.78:0/4225680330 wait complete. 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.103+0000 7f287d855640 1 Processor -- start 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 -- start start 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 0x7f28781989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878106f10 0x7f287819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2878111fc0 con 0x7f2878101de0 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2878111e40 con 0x7f2878102790 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f287d855640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2878112140 con 0x7f2878106f10 2024-09-16T01:10:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42004/0 (socket says 172.21.15.78:42004) 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 -- 172.21.15.78:0/2476433430 learned_addr learned my addr 172.21.15.78:0/2476433430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f28777fe640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878106f10 0x7f287819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f28767fc640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 0x7f28781989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 -- 172.21.15.78:0/2476433430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878106f10 msgr2=0x7f287819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878106f10 0x7f287819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 -- 172.21.15.78:0/2476433430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 msgr2=0x7f28781989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 0x7f28781989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f2876ffd640 1 -- 172.21.15.78:0/2476433430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287819d430 con 0x7f2878101de0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.104+0000 7f28767fc640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 0x7f28781989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.105+0000 7f2876ffd640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f287000e9f0 tx=0x7f287000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.110+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287000cd80 con 0x7f2878101de0 2024-09-16T01:10:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.110+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2870010910 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.110+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287819d720 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.110+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870018990 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.110+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28781a5000 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.111+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28781021c0 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.112+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2870010ac0 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.113+0000 7f287c853640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 0x7f28600a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.113+0000 7f28767fc640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 0x7f28600a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.113+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f28700d0170 con 0x7f2878101de0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.113+0000 7f28767fc640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 0x7f28600a5890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f286c00a300 tx=0x7f286c00a290 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.117+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f287009c970 con 0x7f2878101de0 2024-09-16T01:10:04.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:04 smithi078 ceph-mon[30937]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.283+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2878062630 con 0x7f28600a33d0 2024-09-16T01:10:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.291+0000 7f287c853640 1 -- 172.21.15.78:0/2476433430 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340621 (secure 0 0 0) 0x7f2878062630 con 0x7f28600a33d0 2024-09-16T01:10:04.296 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:10:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 msgr2=0x7f28600a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 0x7f28600a5890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f286c00a300 tx=0x7f286c00a290 comp rx=0 tx=0).stop 2024-09-16T01:10:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 msgr2=0x7f2878198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f287000e9f0 tx=0x7f287000eeb0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 shutdown_connections 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f28600a33d0 0x7f28600a5890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2878106f10 0x7f287819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2878102790 0x7f28781989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 --2- 172.21.15.78:0/2476433430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2878101de0 0x7f2878198480 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.294+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 >> 172.21.15.78:0/2476433430 conn(0x7f28780fd8c0 msgr2=0x7f28780febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.295+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 shutdown_connections 2024-09-16T01:10:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:04.295+0000 7f287d855640 1 -- 172.21.15.78:0/2476433430 wait complete. 2024-09-16T01:10:04.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:04 smithi148 ceph-mon[35030]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:04.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:04 smithi193 ceph-mon[35701]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:05.148 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T01:10:04.015384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656681,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":503,"num_read_kb":374,"num_write":353,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374000,"kb_used_data":12200,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567824,"statfs":{"total":1151940427776,"available":1151557451776,"internally_reserved":0,"allocated":12492800,"data_stored":8048633,"data_compressed":389616,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113170+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:09:39.113170+0000","last_peered":"2024-09-16T01:09:39.113170+0000","last_clean":"2024-09-16T01:09:39.113170+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:09:39.113170+0000","last_undegraded":"2024-09-16T01:09:39.113170+0000","last_fullsized":"2024-09-16T01:09:39.113170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147863+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:09:39.147863+0000","last_peered":"2024-09-16T01:09:39.147863+0000","last_clean":"2024-09-16T01:09:39.147863+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:09:39.147863+0000","last_undegraded":"2024-09-16T01:09:39.147863+0000","last_fullsized":"2024-09-16T01:09:39.147863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110930+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:09:39.110930+0000","last_peered":"2024-09-16T01:09:39.110930+0000","last_clean":"2024-09-16T01:09:39.110930+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:09:39.110930+0000","last_undegraded":"2024-09-16T01:09:39.110930+0000","last_fullsized":"2024-09-16T01:09:39.110930+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148084+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:09:39.148084+0000","last_peered":"2024-09-16T01:09:39.148084+0000","last_clean":"2024-09-16T01:09:39.148084+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:09:39.148084+0000","last_undegraded":"2024-09-16T01:09:39.148084+0000","last_fullsized":"2024-09-16T01:09:39.148084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148378+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:09:39.148378+0000","last_peered":"2024-09-16T01:09:39.148378+0000","last_clean":"2024-09-16T01:09:39.148378+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:09:39.148378+0000","last_undegraded":"2024-09-16T01:09:39.148378+0000","last_fullsized":"2024-09-16T01:09:39.148378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111007+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:09:39.111007+0000","last_peered":"2024-09-16T01:09:39.111007+0000","last_clean":"2024-09-16T01:09:39.111007+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:09:39.111007+0000","last_undegraded":"2024-09-16T01:09:39.111007+0000","last_fullsized":"2024-09-16T01:09:39.111007+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112077+0000","last_change":"2024-09-16T01:08:26.711638+0000","last_active":"2024-09-16T01:09:39.112077+0000","last_peered":"2024-09-16T01:09:39.112077+0000","last_clean":"2024-09-16T01:09:39.112077+0000","last_became_active":"2024-09-16T01:08:26.711038+0000","last_became_peered":"2024-09-16T01:08:26.711038+0000","last_unstale":"2024-09-16T01:09:39.112077+0000","last_undegraded":"2024-09-16T01:09:39.112077+0000","last_fullsized":"2024-09-16T01:09:39.112077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:12.619264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109060+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:09:39.109060+0000","last_peered":"2024-09-16T01:09:39.109060+0000","last_clean":"2024-09-16T01:09:39.109060+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:09:39.109060+0000","last_undegraded":"2024-09-16T01:09:39.109060+0000","last_fullsized":"2024-09-16T01:09:39.109060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111228+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:09:39.111228+0000","last_peered":"2024-09-16T01:09:39.111228+0000","last_clean":"2024-09-16T01:09:39.111228+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:09:39.111228+0000","last_undegraded":"2024-09-16T01:09:39.111228+0000","last_fullsized":"2024-09-16T01:09:39.111228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112881+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:09:39.112881+0000","last_peered":"2024-09-16T01:09:39.112881+0000","last_clean":"2024-09-16T01:09:39.112881+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:09:39.112881+0000","last_undegraded":"2024-09-16T01:09:39.112881+0000","last_fullsized":"2024-09-16T01:09:39.112881+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114931+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:09:39.114931+0000","last_peered":"2024-09-16T01:09:39.114931+0000","last_clean":"2024-09-16T01:09:39.114931+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:09:39.114931+0000","last_undegraded":"2024-09-16T01:09:39.114931+0000","last_fullsized":"2024-09-16T01:09:39.114931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111080+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:09:39.111080+0000","last_peered":"2024-09-16T01:09:39.111080+0000","last_clean":"2024-09-16T01:09:39.111080+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:09:39.111080+0000","last_undegraded":"2024-09-16T01:09:39.111080+0000","last_fullsized":"2024-09-16T01:09:39.111080+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146944+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:09:39.146944+0000","last_peered":"2024-09-16T01:09:39.146944+0000","last_clean":"2024-09-16T01:09:39.146944+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:09:39.146944+0000","last_undegraded":"2024-09-16T01:09:39.146944+0000","last_fullsized":"2024-09-16T01:09:39.146944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110936+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:09:39.110936+0000","last_peered":"2024-09-16T01:09:39.110936+0000","last_clean":"2024-09-16T01:09:39.110936+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:09:39.110936+0000","last_undegraded":"2024-09-16T01:09:39.110936+0000","last_fullsized":"2024-09-16T01:09:39.110936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111508+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:09:39.111508+0000","last_peered":"2024-09-16T01:09:39.111508+0000","last_clean":"2024-09-16T01:09:39.111508+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:09:39.111508+0000","last_undegraded":"2024-09-16T01:09:39.111508+0000","last_fullsized":"2024-09-16T01:09:39.111508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147427+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:09:39.147427+0000","last_peered":"2024-09-16T01:09:39.147427+0000","last_clean":"2024-09-16T01:09:39.147427+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:09:39.147427+0000","last_undegraded":"2024-09-16T01:09:39.147427+0000","last_fullsized":"2024-09-16T01:09:39.147427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111811+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:09:39.111811+0000","last_peered":"2024-09-16T01:09:39.111811+0000","last_clean":"2024-09-16T01:09:39.111811+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:09:39.111811+0000","last_undegraded":"2024-09-16T01:09:39.111811+0000","last_fullsized":"2024-09-16T01:09:39.111811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111496+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:09:39.111496+0000","last_peered":"2024-09-16T01:09:39.111496+0000","last_clean":"2024-09-16T01:09:39.111496+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:09:39.111496+0000","last_undegraded":"2024-09-16T01:09:39.111496+0000","last_fullsized":"2024-09-16T01:09:39.111496+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113684+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:09:39.113684+0000","last_peered":"2024-09-16T01:09:39.113684+0000","last_clean":"2024-09-16T01:09:39.113684+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:09:39.113684+0000","last_undegraded":"2024-09-16T01:09:39.113684+0000","last_fullsized":"2024-09-16T01:09:39.113684+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147516+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:09:39.147516+0000","last_peered":"2024-09-16T01:09:39.147516+0000","last_clean":"2024-09-16T01:09:39.147516+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:09:39.147516+0000","last_undegraded":"2024-09-16T01:09:39.147516+0000","last_fullsized":"2024-09-16T01:09:39.147516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112990+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:09:39.112990+0000","last_peered":"2024-09-16T01:09:39.112990+0000","last_clean":"2024-09-16T01:09:39.112990+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:09:39.112990+0000","last_undegraded":"2024-09-16T01:09:39.112990+0000","last_fullsized":"2024-09-16T01:09:39.112990+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112392+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:09:39.112392+0000","last_peered":"2024-09-16T01:09:39.112392+0000","last_clean":"2024-09-16T01:09:39.112392+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:09:39.112392+0000","last_undegraded":"2024-09-16T01:09:39.112392+0000","last_fullsized":"2024-09-16T01:09:39.112392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109096+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:09:39.109096+0000","last_peered":"2024-09-16T01:09:39.109096+0000","last_clean":"2024-09-16T01:09:39.109096+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:09:39.109096+0000","last_undegraded":"2024-09-16T01:09:39.109096+0000","last_fullsized":"2024-09-16T01:09:39.109096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111693+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:09:39.111693+0000","last_peered":"2024-09-16T01:09:39.111693+0000","last_clean":"2024-09-16T01:09:39.111693+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:09:39.111693+0000","last_undegraded":"2024-09-16T01:09:39.111693+0000","last_fullsized":"2024-09-16T01:09:39.111693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148073+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:09:39.148073+0000","last_peered":"2024-09-16T01:09:39.148073+0000","last_clean":"2024-09-16T01:09:39.148073+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:09:39.148073+0000","last_undegraded":"2024-09-16T01:09:39.148073+0000","last_fullsized":"2024-09-16T01:09:39.148073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112533+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:09:39.112533+0000","last_peered":"2024-09-16T01:09:39.112533+0000","last_clean":"2024-09-16T01:09:39.112533+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:09:39.112533+0000","last_undegraded":"2024-09-16T01:09:39.112533+0000","last_fullsized":"2024-09-16T01:09:39.112533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111140+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:09:39.111140+0000","last_peered":"2024-09-16T01:09:39.111140+0000","last_clean":"2024-09-16T01:09:39.111140+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:09:39.111140+0000","last_undegraded":"2024-09-16T01:09:39.111140+0000","last_fullsized":"2024-09-16T01:09:39.111140+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148167+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:09:39.148167+0000","last_peered":"2024-09-16T01:09:39.148167+0000","last_clean":"2024-09-16T01:09:39.148167+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:09:39.148167+0000","last_undegraded":"2024-09-16T01:09:39.148167+0000","last_fullsized":"2024-09-16T01:09:39.148167+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112996+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:09:39.112996+0000","last_peered":"2024-09-16T01:09:39.112996+0000","last_clean":"2024-09-16T01:09:39.112996+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:09:39.112996+0000","last_undegraded":"2024-09-16T01:09:39.112996+0000","last_fullsized":"2024-09-16T01:09:39.112996+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147127+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:09:39.147127+0000","last_peered":"2024-09-16T01:09:39.147127+0000","last_clean":"2024-09-16T01:09:39.147127+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:09:39.147127+0000","last_undegraded":"2024-09-16T01:09:39.147127+0000","last_fullsized":"2024-09-16T01:09:39.147127+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167691+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:09:39.167691+0000","last_peered":"2024-09-16T01:09:39.167691+0000","last_clean":"2024-09-16T01:09:39.167691+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:09:39.167691+0000","last_undegraded":"2024-09-16T01:09:39.167691+0000","last_fullsized":"2024-09-16T01:09:39.167691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112238+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:09:39.112238+0000","last_peered":"2024-09-16T01:09:39.112238+0000","last_clean":"2024-09-16T01:09:39.112238+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:09:39.112238+0000","last_undegraded":"2024-09-16T01:09:39.112238+0000","last_fullsized":"2024-09-16T01:09:39.112238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114197+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:09:39.114197+0000","last_peered":"2024-09-16T01:09:39.114197+0000","last_clean":"2024-09-16T01:09:39.114197+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:09:39.114197+0000","last_undegraded":"2024-09-16T01:09:39.114197+0000","last_fullsized":"2024-09-16T01:09:39.114197+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114058+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:09:39.114058+0000","last_peered":"2024-09-16T01:09:39.114058+0000","last_clean":"2024-09-16T01:09:39.114058+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:09:39.114058+0000","last_undegraded":"2024-09-16T01:09:39.114058+0000","last_fullsized":"2024-09-16T01:09:39.114058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112855+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:09:39.112855+0000","last_peered":"2024-09-16T01:09:39.112855+0000","last_clean":"2024-09-16T01:09:39.112855+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:09:39.112855+0000","last_undegraded":"2024-09-16T01:09:39.112855+0000","last_fullsized":"2024-09-16T01:09:39.112855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.117797+0000","last_change":"2024-09-16T01:09:39.117797+0000","last_active":"2024-09-16T01:09:39.117797+0000","last_peered":"2024-09-16T01:09:39.117797+0000","last_clean":"2024-09-16T01:09:39.117797+0000","last_became_active":"2024-09-16T01:09:39.117345+0000","last_became_peered":"2024-09-16T01:09:39.117345+0000","last_unstale":"2024-09-16T01:09:39.117797+0000","last_undegraded":"2024-09-16T01:09:39.117797+0000","last_fullsized":"2024-09-16T01:09:39.117797+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:21.410065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148653+0000","last_change":"2024-09-16T01:09:39.148653+0000","last_active":"2024-09-16T01:09:39.148653+0000","last_peered":"2024-09-16T01:09:39.148653+0000","last_clean":"2024-09-16T01:09:39.148653+0000","last_became_active":"2024-09-16T01:09:39.148312+0000","last_became_peered":"2024-09-16T01:09:39.148312+0000","last_unstale":"2024-09-16T01:09:39.148653+0000","last_undegraded":"2024-09-16T01:09:39.148653+0000","last_fullsized":"2024-09-16T01:09:39.148653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:17.031741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147056+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:09:39.147056+0000","last_peered":"2024-09-16T01:09:39.147056+0000","last_clean":"2024-09-16T01:09:39.147056+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:09:39.147056+0000","last_undegraded":"2024-09-16T01:09:39.147056+0000","last_fullsized":"2024-09-16T01:09:39.147056+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167484+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:09:39.167484+0000","last_peered":"2024-09-16T01:09:39.167484+0000","last_clean":"2024-09-16T01:09:39.167484+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:09:39.167484+0000","last_undegraded":"2024-09-16T01:09:39.167484+0000","last_fullsized":"2024-09-16T01:09:39.167484+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114488+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:09:39.114488+0000","last_peered":"2024-09-16T01:09:39.114488+0000","last_clean":"2024-09-16T01:09:39.114488+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:09:39.114488+0000","last_undegraded":"2024-09-16T01:09:39.114488+0000","last_fullsized":"2024-09-16T01:09:39.114488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:10:01.197866+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:10:01.197866+0000","last_peered":"2024-09-16T01:10:01.197866+0000","last_clean":"2024-09-16T01:10:01.197866+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:10:01.197866+0000","last_undegraded":"2024-09-16T01:10:01.197866+0000","last_fullsized":"2024-09-16T01:10:01.197866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167631+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:09:39.167631+0000","last_peered":"2024-09-16T01:09:39.167631+0000","last_clean":"2024-09-16T01:09:39.167631+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:09:39.167631+0000","last_undegraded":"2024-09-16T01:09:39.167631+0000","last_fullsized":"2024-09-16T01:09:39.167631+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167093+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:09:39.167093+0000","last_peered":"2024-09-16T01:09:39.167093+0000","last_clean":"2024-09-16T01:09:39.167093+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:09:39.167093+0000","last_undegraded":"2024-09-16T01:09:39.167093+0000","last_fullsized":"2024-09-16T01:09:39.167093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146926+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:09:39.146926+0000","last_peered":"2024-09-16T01:09:39.146926+0000","last_clean":"2024-09-16T01:09:39.146926+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:09:39.146926+0000","last_undegraded":"2024-09-16T01:09:39.146926+0000","last_fullsized":"2024-09-16T01:09:39.146926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167998+0000","last_change":"2024-09-16T01:09:39.167998+0000","last_active":"2024-09-16T01:09:39.167998+0000","last_peered":"2024-09-16T01:09:39.167998+0000","last_clean":"2024-09-16T01:09:39.167998+0000","last_became_active":"2024-09-16T01:09:39.167533+0000","last_became_peered":"2024-09-16T01:09:39.167533+0000","last_unstale":"2024-09-16T01:09:39.167998+0000","last_undegraded":"2024-09-16T01:09:39.167998+0000","last_fullsized":"2024-09-16T01:09:39.167998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:01.974490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147780+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:09:39.147780+0000","last_peered":"2024-09-16T01:09:39.147780+0000","last_clean":"2024-09-16T01:09:39.147780+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:09:39.147780+0000","last_undegraded":"2024-09-16T01:09:39.147780+0000","last_fullsized":"2024-09-16T01:09:39.147780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112127+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:09:39.112127+0000","last_peered":"2024-09-16T01:09:39.112127+0000","last_clean":"2024-09-16T01:09:39.112127+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:09:39.112127+0000","last_undegraded":"2024-09-16T01:09:39.112127+0000","last_fullsized":"2024-09-16T01:09:39.112127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147052+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:09:39.147052+0000","last_peered":"2024-09-16T01:09:39.147052+0000","last_clean":"2024-09-16T01:09:39.147052+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:09:39.147052+0000","last_undegraded":"2024-09-16T01:09:39.147052+0000","last_fullsized":"2024-09-16T01:09:39.147052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147152+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:09:39.147152+0000","last_peered":"2024-09-16T01:09:39.147152+0000","last_clean":"2024-09-16T01:09:39.147152+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:09:39.147152+0000","last_undegraded":"2024-09-16T01:09:39.147152+0000","last_fullsized":"2024-09-16T01:09:39.147152+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147316+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:09:39.147316+0000","last_peered":"2024-09-16T01:09:39.147316+0000","last_clean":"2024-09-16T01:09:39.147316+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:09:39.147316+0000","last_undegraded":"2024-09-16T01:09:39.147316+0000","last_fullsized":"2024-09-16T01:09:39.147316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112362+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:09:39.112362+0000","last_peered":"2024-09-16T01:09:39.112362+0000","last_clean":"2024-09-16T01:09:39.112362+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:09:39.112362+0000","last_undegraded":"2024-09-16T01:09:39.112362+0000","last_fullsized":"2024-09-16T01:09:39.112362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.108858+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:09:39.108858+0000","last_peered":"2024-09-16T01:09:39.108858+0000","last_clean":"2024-09-16T01:09:39.108858+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:09:39.108858+0000","last_undegraded":"2024-09-16T01:09:39.108858+0000","last_fullsized":"2024-09-16T01:09:39.108858+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147125+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:09:39.147125+0000","last_peered":"2024-09-16T01:09:39.147125+0000","last_clean":"2024-09-16T01:09:39.147125+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:09:39.147125+0000","last_undegraded":"2024-09-16T01:09:39.147125+0000","last_fullsized":"2024-09-16T01:09:39.147125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'1","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115189+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:09:39.115189+0000","last_peered":"2024-09-16T01:09:39.115189+0000","last_clean":"2024-09-16T01:09:39.115189+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:09:39.115189+0000","last_undegraded":"2024-09-16T01:09:39.115189+0000","last_fullsized":"2024-09-16T01:09:39.115189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147445+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:09:39.147445+0000","last_peered":"2024-09-16T01:09:39.147445+0000","last_clean":"2024-09-16T01:09:39.147445+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:09:39.147445+0000","last_undegraded":"2024-09-16T01:09:39.147445+0000","last_fullsized":"2024-09-16T01:09:39.147445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147319+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:09:39.147319+0000","last_peered":"2024-09-16T01:09:39.147319+0000","last_clean":"2024-09-16T01:09:39.147319+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:09:39.147319+0000","last_undegraded":"2024-09-16T01:09:39.147319+0000","last_fullsized":"2024-09-16T01:09:39.147319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167673+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:09:39.167673+0000","last_peered":"2024-09-16T01:09:39.167673+0000","last_clean":"2024-09-16T01:09:39.167673+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:09:39.167673+0000","last_undegraded":"2024-09-16T01:09:39.167673+0000","last_fullsized":"2024-09-16T01:09:39.167673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114298+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:09:39.114298+0000","last_peered":"2024-09-16T01:09:39.114298+0000","last_clean":"2024-09-16T01:09:39.114298+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:09:39.114298+0000","last_undegraded":"2024-09-16T01:09:39.114298+0000","last_fullsized":"2024-09-16T01:09:39.114298+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.119140+0000","last_change":"2024-09-16T01:09:39.119140+0000","last_active":"2024-09-16T01:09:39.119140+0000","last_peered":"2024-09-16T01:09:39.119140+0000","last_clean":"2024-09-16T01:09:39.119140+0000","last_became_active":"2024-09-16T01:09:39.118679+0000","last_became_peered":"2024-09-16T01:09:39.118679+0000","last_unstale":"2024-09-16T01:09:39.119140+0000","last_undegraded":"2024-09-16T01:09:39.119140+0000","last_fullsized":"2024-09-16T01:09:39.119140+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:14.355461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111730+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:09:39.111730+0000","last_peered":"2024-09-16T01:09:39.111730+0000","last_clean":"2024-09-16T01:09:39.111730+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:09:39.111730+0000","last_undegraded":"2024-09-16T01:09:39.111730+0000","last_fullsized":"2024-09-16T01:09:39.111730+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115575+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:09:39.115575+0000","last_peered":"2024-09-16T01:09:39.115575+0000","last_clean":"2024-09-16T01:09:39.115575+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:09:39.115575+0000","last_undegraded":"2024-09-16T01:09:39.115575+0000","last_fullsized":"2024-09-16T01:09:39.115575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121178+0000","last_change":"2024-09-16T01:09:39.121178+0000","last_active":"2024-09-16T01:09:39.121178+0000","last_peered":"2024-09-16T01:09:39.121178+0000","last_clean":"2024-09-16T01:09:39.121178+0000","last_became_active":"2024-09-16T01:09:39.120781+0000","last_became_peered":"2024-09-16T01:09:39.120781+0000","last_unstale":"2024-09-16T01:09:39.121178+0000","last_undegraded":"2024-09-16T01:09:39.121178+0000","last_fullsized":"2024-09-16T01:09:39.121178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:43.670325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.110770+0000","last_change":"2024-09-16T01:08:26.956123+0000","last_active":"2024-09-16T01:09:39.110770+0000","last_peered":"2024-09-16T01:09:39.110770+0000","last_clean":"2024-09-16T01:09:39.110770+0000","last_became_active":"2024-09-16T01:08:26.955746+0000","last_became_peered":"2024-09-16T01:08:26.955746+0000","last_unstale":"2024-09-16T01:09:39.110770+0000","last_undegraded":"2024-09-16T01:09:39.110770+0000","last_fullsized":"2024-09-16T01:09:39.110770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:09:40.575736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167656+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:09:39.167656+0000","last_peered":"2024-09-16T01:09:39.167656+0000","last_clean":"2024-09-16T01:09:39.167656+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:09:39.167656+0000","last_undegraded":"2024-09-16T01:09:39.167656+0000","last_fullsized":"2024-09-16T01:09:39.167656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146816+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:09:39.146816+0000","last_peered":"2024-09-16T01:09:39.146816+0000","last_clean":"2024-09-16T01:09:39.146816+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:09:39.146816+0000","last_undegraded":"2024-09-16T01:09:39.146816+0000","last_fullsized":"2024-09-16T01:09:39.146816+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":96,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:59.195595+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:09:59.195595+0000","last_peered":"2024-09-16T01:09:59.195595+0000","last_clean":"2024-09-16T01:09:59.195595+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:09:59.195595+0000","last_undegraded":"2024-09-16T01:09:59.195595+0000","last_fullsized":"2024-09-16T01:09:59.195595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113069+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:09:39.113069+0000","last_peered":"2024-09-16T01:09:39.113069+0000","last_clean":"2024-09-16T01:09:39.113069+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:09:39.113069+0000","last_undegraded":"2024-09-16T01:09:39.113069+0000","last_fullsized":"2024-09-16T01:09:39.113069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147170+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:09:39.147170+0000","last_peered":"2024-09-16T01:09:39.147170+0000","last_clean":"2024-09-16T01:09:39.147170+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:09:39.147170+0000","last_undegraded":"2024-09-16T01:09:39.147170+0000","last_fullsized":"2024-09-16T01:09:39.147170+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115237+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:09:39.115237+0000","last_peered":"2024-09-16T01:09:39.115237+0000","last_clean":"2024-09-16T01:09:39.115237+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:09:39.115237+0000","last_undegraded":"2024-09-16T01:09:39.115237+0000","last_fullsized":"2024-09-16T01:09:39.115237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114958+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:09:39.114958+0000","last_peered":"2024-09-16T01:09:39.114958+0000","last_clean":"2024-09-16T01:09:39.114958+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:09:39.114958+0000","last_undegraded":"2024-09-16T01:09:39.114958+0000","last_fullsized":"2024-09-16T01:09:39.114958+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167295+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:09:39.167295+0000","last_peered":"2024-09-16T01:09:39.167295+0000","last_clean":"2024-09-16T01:09:39.167295+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:09:39.167295+0000","last_undegraded":"2024-09-16T01:09:39.167295+0000","last_fullsized":"2024-09-16T01:09:39.167295+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111749+0000","last_change":"2024-09-16T01:08:26.710927+0000","last_active":"2024-09-16T01:09:39.111749+0000","last_peered":"2024-09-16T01:09:39.111749+0000","last_clean":"2024-09-16T01:09:39.111749+0000","last_became_active":"2024-09-16T01:08:26.709796+0000","last_became_peered":"2024-09-16T01:08:26.709796+0000","last_unstale":"2024-09-16T01:09:39.111749+0000","last_undegraded":"2024-09-16T01:09:39.111749+0000","last_fullsized":"2024-09-16T01:09:39.111749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:30.270120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114817+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:09:39.114817+0000","last_peered":"2024-09-16T01:09:39.114817+0000","last_clean":"2024-09-16T01:09:39.114817+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:09:39.114817+0000","last_undegraded":"2024-09-16T01:09:39.114817+0000","last_fullsized":"2024-09-16T01:09:39.114817+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111895+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:09:39.111895+0000","last_peered":"2024-09-16T01:09:39.111895+0000","last_clean":"2024-09-16T01:09:39.111895+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:09:39.111895+0000","last_undegraded":"2024-09-16T01:09:39.111895+0000","last_fullsized":"2024-09-16T01:09:39.111895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:10:02.200219+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:10:02.200219+0000","last_peered":"2024-09-16T01:10:02.200219+0000","last_clean":"2024-09-16T01:10:02.200219+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:10:02.200219+0000","last_undegraded":"2024-09-16T01:10:02.200219+0000","last_fullsized":"2024-09-16T01:10:02.200219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62076,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":123,"num_read_kb":122,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121127+0000","last_change":"2024-09-16T01:09:39.121127+0000","last_active":"2024-09-16T01:09:39.121127+0000","last_peered":"2024-09-16T01:09:39.121127+0000","last_clean":"2024-09-16T01:09:39.121127+0000","last_became_active":"2024-09-16T01:09:39.120736+0000","last_became_peered":"2024-09-16T01:09:39.120736+0000","last_unstale":"2024-09-16T01:09:39.121127+0000","last_undegraded":"2024-09-16T01:09:39.121127+0000","last_fullsized":"2024-09-16T01:09:39.121127+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:28.281410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.168566+0000","last_change":"2024-09-16T01:09:39.168566+0000","last_active":"2024-09-16T01:09:39.168566+0000","last_peered":"2024-09-16T01:09:39.168566+0000","last_clean":"2024-09-16T01:09:39.168566+0000","last_became_active":"2024-09-16T01:09:39.168127+0000","last_became_peered":"2024-09-16T01:09:39.168127+0000","last_unstale":"2024-09-16T01:09:39.168566+0000","last_undegraded":"2024-09-16T01:09:39.168566+0000","last_fullsized":"2024-09-16T01:09:39.168566+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:52.054331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115057+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:09:39.115057+0000","last_peered":"2024-09-16T01:09:39.115057+0000","last_clean":"2024-09-16T01:09:39.115057+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:09:39.115057+0000","last_undegraded":"2024-09-16T01:09:39.115057+0000","last_fullsized":"2024-09-16T01:09:39.115057+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121335+0000","last_change":"2024-09-16T01:09:39.121335+0000","last_active":"2024-09-16T01:09:39.121335+0000","last_peered":"2024-09-16T01:09:39.121335+0000","last_clean":"2024-09-16T01:09:39.121335+0000","last_became_active":"2024-09-16T01:09:39.120738+0000","last_became_peered":"2024-09-16T01:09:39.120738+0000","last_unstale":"2024-09-16T01:09:39.121335+0000","last_undegraded":"2024-09-16T01:09:39.121335+0000","last_fullsized":"2024-09-16T01:09:39.121335+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:09.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148388+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:09:39.148388+0000","last_peered":"2024-09-16T01:09:39.148388+0000","last_clean":"2024-09-16T01:09:39.148388+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:09:39.148388+0000","last_undegraded":"2024-09-16T01:09:39.148388+0000","last_fullsized":"2024-09-16T01:09:39.148388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147219+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:09:39.147219+0000","last_peered":"2024-09-16T01:09:39.147219+0000","last_clean":"2024-09-16T01:09:39.147219+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:09:39.147219+0000","last_undegraded":"2024-09-16T01:09:39.147219+0000","last_fullsized":"2024-09-16T01:09:39.147219+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112418+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:09:39.112418+0000","last_peered":"2024-09-16T01:09:39.112418+0000","last_clean":"2024-09-16T01:09:39.112418+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:09:39.112418+0000","last_undegraded":"2024-09-16T01:09:39.112418+0000","last_fullsized":"2024-09-16T01:09:39.112418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109375+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:09:39.109375+0000","last_peered":"2024-09-16T01:09:39.109375+0000","last_clean":"2024-09-16T01:09:39.109375+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:09:39.109375+0000","last_undegraded":"2024-09-16T01:09:39.109375+0000","last_fullsized":"2024-09-16T01:09:39.109375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:10:01.198602+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:10:01.198602+0000","last_peered":"2024-09-16T01:10:01.198602+0000","last_clean":"2024-09-16T01:10:01.198602+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:10:01.198602+0000","last_undegraded":"2024-09-16T01:10:01.198602+0000","last_fullsized":"2024-09-16T01:10:01.198602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147270+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:09:39.147270+0000","last_peered":"2024-09-16T01:09:39.147270+0000","last_clean":"2024-09-16T01:09:39.147270+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:09:39.147270+0000","last_undegraded":"2024-09-16T01:09:39.147270+0000","last_fullsized":"2024-09-16T01:09:39.147270+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147905+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:09:39.147905+0000","last_peered":"2024-09-16T01:09:39.147905+0000","last_clean":"2024-09-16T01:09:39.147905+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:09:39.147905+0000","last_undegraded":"2024-09-16T01:09:39.147905+0000","last_fullsized":"2024-09-16T01:09:39.147905+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112495+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:09:39.112495+0000","last_peered":"2024-09-16T01:09:39.112495+0000","last_clean":"2024-09-16T01:09:39.112495+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:09:39.112495+0000","last_undegraded":"2024-09-16T01:09:39.112495+0000","last_fullsized":"2024-09-16T01:09:39.112495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114701+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:09:39.114701+0000","last_peered":"2024-09-16T01:09:39.114701+0000","last_clean":"2024-09-16T01:09:39.114701+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:09:39.114701+0000","last_undegraded":"2024-09-16T01:09:39.114701+0000","last_fullsized":"2024-09-16T01:09:39.114701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114341+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:09:39.114341+0000","last_peered":"2024-09-16T01:09:39.114341+0000","last_clean":"2024-09-16T01:09:39.114341+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:09:39.114341+0000","last_undegraded":"2024-09-16T01:09:39.114341+0000","last_fullsized":"2024-09-16T01:09:39.114341+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147328+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:09:39.147328+0000","last_peered":"2024-09-16T01:09:39.147328+0000","last_clean":"2024-09-16T01:09:39.147328+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:09:39.147328+0000","last_undegraded":"2024-09-16T01:09:39.147328+0000","last_fullsized":"2024-09-16T01:09:39.147328+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167078+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:09:39.167078+0000","last_peered":"2024-09-16T01:09:39.167078+0000","last_clean":"2024-09-16T01:09:39.167078+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:09:39.167078+0000","last_undegraded":"2024-09-16T01:09:39.167078+0000","last_fullsized":"2024-09-16T01:09:39.167078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114306+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:09:39.114306+0000","last_peered":"2024-09-16T01:09:39.114306+0000","last_clean":"2024-09-16T01:09:39.114306+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:09:39.114306+0000","last_undegraded":"2024-09-16T01:09:39.114306+0000","last_fullsized":"2024-09-16T01:09:39.114306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114765+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:09:39.114765+0000","last_peered":"2024-09-16T01:09:39.114765+0000","last_clean":"2024-09-16T01:09:39.114765+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:09:39.114765+0000","last_undegraded":"2024-09-16T01:09:39.114765+0000","last_fullsized":"2024-09-16T01:09:39.114765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111232+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:09:39.111232+0000","last_peered":"2024-09-16T01:09:39.111232+0000","last_clean":"2024-09-16T01:09:39.111232+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:09:39.111232+0000","last_undegraded":"2024-09-16T01:09:39.111232+0000","last_fullsized":"2024-09-16T01:09:39.111232+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":97,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:10:00.197765+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:10:00.197765+0000","last_peered":"2024-09-16T01:10:00.197765+0000","last_clean":"2024-09-16T01:10:00.197765+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:10:00.197765+0000","last_undegraded":"2024-09-16T01:10:00.197765+0000","last_fullsized":"2024-09-16T01:10:00.197765+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112366+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:09:39.112366+0000","last_peered":"2024-09-16T01:09:39.112366+0000","last_clean":"2024-09-16T01:09:39.112366+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:09:39.112366+0000","last_undegraded":"2024-09-16T01:09:39.112366+0000","last_fullsized":"2024-09-16T01:09:39.112366+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111619+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:09:39.111619+0000","last_peered":"2024-09-16T01:09:39.111619+0000","last_clean":"2024-09-16T01:09:39.111619+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:09:39.111619+0000","last_undegraded":"2024-09-16T01:09:39.111619+0000","last_fullsized":"2024-09-16T01:09:39.111619+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.119041+0000","last_change":"2024-09-16T01:09:39.119041+0000","last_active":"2024-09-16T01:09:39.119041+0000","last_peered":"2024-09-16T01:09:39.119041+0000","last_clean":"2024-09-16T01:09:39.119041+0000","last_became_active":"2024-09-16T01:09:39.118531+0000","last_became_peered":"2024-09-16T01:09:39.118531+0000","last_unstale":"2024-09-16T01:09:39.119041+0000","last_undegraded":"2024-09-16T01:09:39.119041+0000","last_fullsized":"2024-09-16T01:09:39.119041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:32:18.412355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112004+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:09:39.112004+0000","last_peered":"2024-09-16T01:09:39.112004+0000","last_clean":"2024-09-16T01:09:39.112004+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:09:39.112004+0000","last_undegraded":"2024-09-16T01:09:39.112004+0000","last_fullsized":"2024-09-16T01:09:39.112004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111015+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:09:39.111015+0000","last_peered":"2024-09-16T01:09:39.111015+0000","last_clean":"2024-09-16T01:09:39.111015+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:09:39.111015+0000","last_undegraded":"2024-09-16T01:09:39.111015+0000","last_fullsized":"2024-09-16T01:09:39.111015+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147535+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:09:39.147535+0000","last_peered":"2024-09-16T01:09:39.147535+0000","last_clean":"2024-09-16T01:09:39.147535+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:09:39.147535+0000","last_undegraded":"2024-09-16T01:09:39.147535+0000","last_fullsized":"2024-09-16T01:09:39.147535+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167637+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:09:39.167637+0000","last_peered":"2024-09-16T01:09:39.167637+0000","last_clean":"2024-09-16T01:09:39.167637+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:09:39.167637+0000","last_undegraded":"2024-09-16T01:09:39.167637+0000","last_fullsized":"2024-09-16T01:09:39.167637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.113887+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:09:39.113887+0000","last_peered":"2024-09-16T01:09:39.113887+0000","last_clean":"2024-09-16T01:09:39.113887+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:09:39.113887+0000","last_undegraded":"2024-09-16T01:09:39.113887+0000","last_fullsized":"2024-09-16T01:09:39.113887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112241+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:09:39.112241+0000","last_peered":"2024-09-16T01:09:39.112241+0000","last_clean":"2024-09-16T01:09:39.112241+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:09:39.112241+0000","last_undegraded":"2024-09-16T01:09:39.112241+0000","last_fullsized":"2024-09-16T01:09:39.112241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.148075+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:09:39.148075+0000","last_peered":"2024-09-16T01:09:39.148075+0000","last_clean":"2024-09-16T01:09:39.148075+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:09:39.148075+0000","last_undegraded":"2024-09-16T01:09:39.148075+0000","last_fullsized":"2024-09-16T01:09:39.148075+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.121080+0000","last_change":"2024-09-16T01:09:39.121080+0000","last_active":"2024-09-16T01:09:39.121080+0000","last_peered":"2024-09-16T01:09:39.121080+0000","last_clean":"2024-09-16T01:09:39.121080+0000","last_became_active":"2024-09-16T01:09:39.120707+0000","last_became_peered":"2024-09-16T01:09:39.120707+0000","last_unstale":"2024-09-16T01:09:39.121080+0000","last_undegraded":"2024-09-16T01:09:39.121080+0000","last_fullsized":"2024-09-16T01:09:39.121080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:08.521558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147425+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:09:39.147425+0000","last_peered":"2024-09-16T01:09:39.147425+0000","last_clean":"2024-09-16T01:09:39.147425+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:09:39.147425+0000","last_undegraded":"2024-09-16T01:09:39.147425+0000","last_fullsized":"2024-09-16T01:09:39.147425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109196+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:09:39.109196+0000","last_peered":"2024-09-16T01:09:39.109196+0000","last_clean":"2024-09-16T01:09:39.109196+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:09:39.109196+0000","last_undegraded":"2024-09-16T01:09:39.109196+0000","last_fullsized":"2024-09-16T01:09:39.109196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167487+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:09:39.167487+0000","last_peered":"2024-09-16T01:09:39.167487+0000","last_clean":"2024-09-16T01:09:39.167487+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:09:39.167487+0000","last_undegraded":"2024-09-16T01:09:39.167487+0000","last_fullsized":"2024-09-16T01:09:39.167487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.115061+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:09:39.115061+0000","last_peered":"2024-09-16T01:09:39.115061+0000","last_clean":"2024-09-16T01:09:39.115061+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:09:39.115061+0000","last_undegraded":"2024-09-16T01:09:39.115061+0000","last_fullsized":"2024-09-16T01:09:39.115061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111783+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:09:39.111783+0000","last_peered":"2024-09-16T01:09:39.111783+0000","last_clean":"2024-09-16T01:09:39.111783+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:09:39.111783+0000","last_undegraded":"2024-09-16T01:09:39.111783+0000","last_fullsized":"2024-09-16T01:09:39.111783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.112954+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:09:39.112954+0000","last_peered":"2024-09-16T01:09:39.112954+0000","last_clean":"2024-09-16T01:09:39.112954+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:09:39.112954+0000","last_undegraded":"2024-09-16T01:09:39.112954+0000","last_fullsized":"2024-09-16T01:09:39.112954+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114746+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:09:39.114746+0000","last_peered":"2024-09-16T01:09:39.114746+0000","last_clean":"2024-09-16T01:09:39.114746+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:09:39.114746+0000","last_undegraded":"2024-09-16T01:09:39.114746+0000","last_fullsized":"2024-09-16T01:09:39.114746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.108831+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:09:39.108831+0000","last_peered":"2024-09-16T01:09:39.108831+0000","last_clean":"2024-09-16T01:09:39.108831+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:09:39.108831+0000","last_undegraded":"2024-09-16T01:09:39.108831+0000","last_fullsized":"2024-09-16T01:09:39.108831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147754+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:09:39.147754+0000","last_peered":"2024-09-16T01:09:39.147754+0000","last_clean":"2024-09-16T01:09:39.147754+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:09:39.147754+0000","last_undegraded":"2024-09-16T01:09:39.147754+0000","last_fullsized":"2024-09-16T01:09:39.147754+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147372+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:09:39.147372+0000","last_peered":"2024-09-16T01:09:39.147372+0000","last_clean":"2024-09-16T01:09:39.147372+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:09:39.147372+0000","last_undegraded":"2024-09-16T01:09:39.147372+0000","last_fullsized":"2024-09-16T01:09:39.147372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.167450+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:09:39.167450+0000","last_peered":"2024-09-16T01:09:39.167450+0000","last_clean":"2024-09-16T01:09:39.167450+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:09:39.167450+0000","last_undegraded":"2024-09-16T01:09:39.167450+0000","last_fullsized":"2024-09-16T01:09:39.167450+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111871+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:09:39.111871+0000","last_peered":"2024-09-16T01:09:39.111871+0000","last_clean":"2024-09-16T01:09:39.111871+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:09:39.111871+0000","last_undegraded":"2024-09-16T01:09:39.111871+0000","last_fullsized":"2024-09-16T01:09:39.111871+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147313+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:09:39.147313+0000","last_peered":"2024-09-16T01:09:39.147313+0000","last_clean":"2024-09-16T01:09:39.147313+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:09:39.147313+0000","last_undegraded":"2024-09-16T01:09:39.147313+0000","last_fullsized":"2024-09-16T01:09:39.147313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114426+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:09:39.114426+0000","last_peered":"2024-09-16T01:09:39.114426+0000","last_clean":"2024-09-16T01:09:39.114426+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:09:39.114426+0000","last_undegraded":"2024-09-16T01:09:39.114426+0000","last_fullsized":"2024-09-16T01:09:39.114426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.109572+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:09:39.109572+0000","last_peered":"2024-09-16T01:09:39.109572+0000","last_clean":"2024-09-16T01:09:39.109572+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:09:39.109572+0000","last_undegraded":"2024-09-16T01:09:39.109572+0000","last_fullsized":"2024-09-16T01:09:39.109572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147238+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:09:39.147238+0000","last_peered":"2024-09-16T01:09:39.147238+0000","last_clean":"2024-09-16T01:09:39.147238+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:09:39.147238+0000","last_undegraded":"2024-09-16T01:09:39.147238+0000","last_fullsized":"2024-09-16T01:09:39.147238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.146902+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:09:39.146902+0000","last_peered":"2024-09-16T01:09:39.146902+0000","last_clean":"2024-09-16T01:09:39.146902+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:09:39.146902+0000","last_undegraded":"2024-09-16T01:09:39.146902+0000","last_fullsized":"2024-09-16T01:09:39.146902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114244+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:09:39.114244+0000","last_peered":"2024-09-16T01:09:39.114244+0000","last_clean":"2024-09-16T01:09:39.114244+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:09:39.114244+0000","last_undegraded":"2024-09-16T01:09:39.114244+0000","last_fullsized":"2024-09-16T01:09:39.114244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111065+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:09:39.111065+0000","last_peered":"2024-09-16T01:09:39.111065+0000","last_clean":"2024-09-16T01:09:39.111065+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:09:39.111065+0000","last_undegraded":"2024-09-16T01:09:39.111065+0000","last_fullsized":"2024-09-16T01:09:39.111065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.111750+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:09:39.111750+0000","last_peered":"2024-09-16T01:09:39.111750+0000","last_clean":"2024-09-16T01:09:39.111750+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:09:39.111750+0000","last_undegraded":"2024-09-16T01:09:39.111750+0000","last_fullsized":"2024-09-16T01:09:39.111750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147807+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:09:39.147807+0000","last_peered":"2024-09-16T01:09:39.147807+0000","last_clean":"2024-09-16T01:09:39.147807+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:09:39.147807+0000","last_undegraded":"2024-09-16T01:09:39.147807+0000","last_fullsized":"2024-09-16T01:09:39.147807+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.147988+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:09:39.147988+0000","last_peered":"2024-09-16T01:09:39.147988+0000","last_clean":"2024-09-16T01:09:39.147988+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:09:39.147988+0000","last_undegraded":"2024-09-16T01:09:39.147988+0000","last_fullsized":"2024-09-16T01:09:39.147988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T01:09:39.114641+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:09:39.114641+0000","last_peered":"2024-09-16T01:09:39.114641+0000","last_clean":"2024-09-16T01:09:39.114641+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:09:39.114641+0000","last_undegraded":"2024-09-16T01:09:39.114641+0000","last_fullsized":"2024-09-16T01:09:39.114641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66079,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":367,"num_read_kb":255,"num_write":170,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282755,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481825,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":999424,"data_stored":508555,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710758,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1142784,"data_stored":1158292,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874283,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1019904,"data_stored":512887,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103215,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1015808,"data_stored":508711,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1064960,"data_stored":1104622,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528377,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":503967,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1073152,"data_stored":1099817,"data_compressed":59444,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":509010,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":504161,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":567303,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1060864,"data_stored":565953,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1019904,"data_stored":505355,"data_compressed":23476,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34305,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:10:05.155 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T01:10:05.155 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T01:10:05.155 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-16T01:10:05.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-16T01:10:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:05 smithi148 ceph-mon[35030]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:05.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:05 smithi078 ceph-mon[30937]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:05 smithi193 ceph-mon[35701]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:05.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 -- 172.21.15.78:0/280450831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc106a80 msgr2=0x7f46bc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 --2- 172.21.15.78:0/280450831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc106a80 0x7f46bc106e60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009930 tx=0x7f46b402a420 comp rx=0 tx=0).stop 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 -- 172.21.15.78:0/280450831 shutdown_connections 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 --2- 172.21.15.78:0/280450831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc101420 0x7f46bc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 --2- 172.21.15.78:0/280450831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc100a80 0x7f46bc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 --2- 172.21.15.78:0/280450831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc106a80 0x7f46bc106e60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 -- 172.21.15.78:0/280450831 >> 172.21.15.78:0/280450831 conn(0x7f46bc0fc7a0 msgr2=0x7f46bc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 -- 172.21.15.78:0/280450831 shutdown_connections 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 -- 172.21.15.78:0/280450831 wait complete. 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.894+0000 7f46c4355640 1 Processor -- start 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 -- start start 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc100a80 0x7f46bc0fce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc106a80 0x7f46bc0fd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc110ea0 con 0x7f46bc101420 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc110d20 con 0x7f46bc106a80 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c4355640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc111020 con 0x7f46bc100a80 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42028/0 (socket says 172.21.15.78:42028) 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 -- 172.21.15.78:0/13763585 learned_addr learned my addr 172.21.15.78:0/13763585 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c20f1640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc100a80 0x7f46bc0fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 -- 172.21.15.78:0/13763585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc100a80 msgr2=0x7f46bc0fce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc100a80 0x7f46bc0fce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 -- 172.21.15.78:0/13763585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc106a80 msgr2=0x7f46bc0fd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c28f2640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc106a80 0x7f46bc0fd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc106a80 0x7f46bc0fd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 -- 172.21.15.78:0/13763585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc1a69d0 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46c18f0640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b4a0 tx=0x7f46b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.895+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0014070 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.896+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc1a6cc0 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.896+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46b0010af0 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.896+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0017d10 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.896+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc1a71f0 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.897+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc100ee0 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.898+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f46b0017070 con 0x7f46bc101420 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.898+0000 7f46bb7fe640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 0x7f46a40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.898+0000 7f46c20f1640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 0x7f46a40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.899+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f46b00cf110 con 0x7f46bc101420 2024-09-16T01:10:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.899+0000 7f46c20f1640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 0x7f46a40a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b4031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:05.903+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f46b009ba80 con 0x7f46bc101420 2024-09-16T01:10:06.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.102+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f46bc10a000 con 0x7f46bc101420 2024-09-16T01:10:06.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.103+0000 7f46bb7fe640 1 -- 172.21.15.78:0/13763585 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f46b009b380 con 0x7f46bc101420 2024-09-16T01:10:06.104 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:06.104 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 msgr2=0x7f46a40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 0x7f46a40a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b4031490 comp rx=0 tx=0).stop 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 msgr2=0x7f46bc0fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b4a0 tx=0x7f46b000b960 comp rx=0 tx=0).stop 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 shutdown_connections 2024-09-16T01:10:06.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f46a40a32d0 0x7f46a40a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46bc106a80 0x7f46bc0fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46bc101420 0x7f46bc0fd390 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 --2- 172.21.15.78:0/13763585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46bc100a80 0x7f46bc0fce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 >> 172.21.15.78:0/13763585 conn(0x7f46bc0fc7a0 msgr2=0x7f46bc06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 shutdown_connections 2024-09-16T01:10:06.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:06.105+0000 7f46c4355640 1 -- 172.21.15.78:0/13763585 wait complete. 2024-09-16T01:10:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:06 smithi148 ceph-mon[35030]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:06 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/13763585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T01:10:06.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:06 smithi078 ceph-mon[30937]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:06.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:06 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/13763585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T01:10:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:06 smithi193 ceph-mon[35701]: from='client.15006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:06 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/13763585' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-16T01:10:06.909 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-16T01:10:06.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:07.166 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:07 smithi078 ceph-mon[30937]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:07 smithi148 ceph-mon[35030]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:07 smithi193 ceph-mon[35701]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:07.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 -- 172.21.15.78:0/2842959735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0106f00 msgr2=0x7f71d010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:07.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 --2- 172.21.15.78:0/2842959735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0106f00 0x7f71d010ea80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009ca0 tx=0x7f71c002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 -- 172.21.15.78:0/2842959735 shutdown_connections 2024-09-16T01:10:07.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 --2- 172.21.15.78:0/2842959735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0106f00 0x7f71d010ea80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 --2- 172.21.15.78:0/2842959735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 --2- 172.21.15.78:0/2842959735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0101dd0 0x7f71d01021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 -- 172.21.15.78:0/2842959735 >> 172.21.15.78:0/2842959735 conn(0x7f71d00fd8b0 msgr2=0x7f71d00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 -- 172.21.15.78:0/2842959735 shutdown_connections 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.655+0000 7f71d8560640 1 -- 172.21.15.78:0/2842959735 wait complete. 2024-09-16T01:10:07.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 Processor -- start 2024-09-16T01:10:07.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 -- start start 2024-09-16T01:10:07.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:07.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:07.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0106f00 0x7f71d019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:07.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0111f60 con 0x7f71d0101dd0 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f71d0111de0 con 0x7f71d0102780 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d8560640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71d01120e0 con 0x7f71d0106f00 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42046/0 (socket says 172.21.15.78:42046) 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 -- 172.21.15.78:0/2712151655 learned_addr learned my addr 172.21.15.78:0/2712151655 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:07.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d5afb640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d6afd640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0106f00 0x7f71d019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 -- 172.21.15.78:0/2712151655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0106f00 msgr2=0x7f71d019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0106f00 0x7f71d019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 -- 172.21.15.78:0/2712151655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 msgr2=0x7f71d01988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:07.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 -- 172.21.15.78:0/2712151655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d019d360 con 0x7f71d0101dd0 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d5afb640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.656+0000 7f71d62fc640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f71c800c970 tx=0x7f71c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.657+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8007bb0 con 0x7f71d0101dd0 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.657+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d019d650 con 0x7f71d0101dd0 2024-09-16T01:10:07.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.657+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71c8007d50 con 0x7f71d0101dd0 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.657+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c800f450 con 0x7f71d0101dd0 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.657+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d01a4f30 con 0x7f71d0101dd0 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.658+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d01021b0 con 0x7f71d0101dd0 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.659+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f71c8004fe0 con 0x7f71d0101dd0 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.659+0000 7f71cf7fe640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 0x7f71b80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:07.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.659+0000 7f71d5afb640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 0x7f71b80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:07.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.660+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f71c80d0080 con 0x7f71d0101dd0 2024-09-16T01:10:07.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.660+0000 7f71d5afb640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 0x7f71b80a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f71c4005fd0 tx=0x7f71c4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:07.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.663+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f71c8014030 con 0x7f71d0101dd0 2024-09-16T01:10:07.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.865+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71d010a140 con 0x7f71d0101dd0 2024-09-16T01:10:07.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.866+0000 7f71cf7fe640 1 -- 172.21.15.78:0/2712151655 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7f71c80bae20 con 0x7f71d0101dd0 2024-09-16T01:10:07.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:07.867 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:10:07.867 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 msgr2=0x7f71b80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 0x7f71b80a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f71c4005fd0 tx=0x7f71c4005850 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 msgr2=0x7f71d01983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f71c800c970 tx=0x7f71c800ce30 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 shutdown_connections 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f71b80a32d0 0x7f71b80a5790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71d0106f00 0x7f71d019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71d0102780 0x7f71d01988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 --2- 172.21.15.78:0/2712151655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71d0101dd0 0x7f71d01983b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 >> 172.21.15.78:0/2712151655 conn(0x7f71d00fd8b0 msgr2=0x7f71d00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:07.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 shutdown_connections 2024-09-16T01:10:07.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:07.868+0000 7f71d8560640 1 -- 172.21.15.78:0/2712151655 wait complete. 2024-09-16T01:10:08.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:08 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2712151655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:08.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2712151655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:08 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2712151655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:08.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:08.836 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:09.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:09 smithi078 ceph-mon[30937]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 -- 172.21.15.78:0/1885160976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f28102da0 msgr2=0x7f7f28103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 --2- 172.21.15.78:0/1885160976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f28102da0 0x7f7f28103180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009900 tx=0x7f7f2402a480 comp rx=0 tx=0).stop 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 -- 172.21.15.78:0/1885160976 shutdown_connections 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 --2- 172.21.15.78:0/1885160976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f2810c180 0x7f7f2810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 --2- 172.21.15.78:0/1885160976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f281036c0 0x7f7f2810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 --2- 172.21.15.78:0/1885160976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f28102da0 0x7f7f28103180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 -- 172.21.15.78:0/1885160976 >> 172.21.15.78:0/1885160976 conn(0x7f7f280fc7a0 msgr2=0x7f7f280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 -- 172.21.15.78:0/1885160976 shutdown_connections 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.339+0000 7f7f3064f640 1 -- 172.21.15.78:0/1885160976 wait complete. 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 Processor -- start 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 -- start start 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f28102da0 0x7f7f28198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f2810c180 0x7f7f2819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28111fa0 con 0x7f7f281036c0 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28111e20 con 0x7f7f28102da0 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f3064f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28112120 con 0x7f7f2810c180 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42076/0 (socket says 172.21.15.78:42076) 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 -- 172.21.15.78:0/319486852 learned_addr learned my addr 172.21.15.78:0/319486852 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2ebec640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f2810c180 0x7f7f2819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2e3eb640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f28102da0 0x7f7f28198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 -- 172.21.15.78:0/319486852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f2810c180 msgr2=0x7f7f2819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:09.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f2810c180 0x7f7f2819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 -- 172.21.15.78:0/319486852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f28102da0 msgr2=0x7f7f28198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f28102da0 0x7f7f28198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.340+0000 7f7f2dbea640 1 -- 172.21.15.78:0/319486852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2819d3d0 con 0x7f7f281036c0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f2dbea640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b4a0 tx=0x7f7f1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18014070 con 0x7f7f281036c0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f18010af0 con 0x7f7f281036c0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f2819d6c0 con 0x7f7f281036c0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18017d10 con 0x7f7f281036c0 2024-09-16T01:10:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.341+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f281a4fa0 con 0x7f7f281036c0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.342+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f28103180 con 0x7f7f281036c0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.343+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7f18017070 con 0x7f7f281036c0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.344+0000 7f7f1f7fe640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 0x7f7f140a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.344+0000 7f7f2e3eb640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 0x7f7f140a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.344+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f180cf500 con 0x7f7f281036c0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.344+0000 7f7f2e3eb640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 0x7f7f140a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f240096c0 tx=0x7f7f24031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.347+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7f1809be70 con 0x7f7f281036c0 2024-09-16T01:10:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:09 smithi193 ceph-mon[35701]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:09.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.546+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f28199780 con 0x7f7f281036c0 2024-09-16T01:10:09.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.547+0000 7f7f1f7fe640 1 -- 172.21.15.78:0/319486852 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7f7f28199780 con 0x7f7f281036c0 2024-09-16T01:10:09.548 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:09.548 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:10:09.548 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 msgr2=0x7f7f140a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 0x7f7f140a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f240096c0 tx=0x7f7f24031490 comp rx=0 tx=0).stop 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 msgr2=0x7f7f28198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b4a0 tx=0x7f7f1800b960 comp rx=0 tx=0).stop 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 shutdown_connections 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7f140a32d0 0x7f7f140a5790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f2810c180 0x7f7f2819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f281036c0 0x7f7f28198960 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 --2- 172.21.15.78:0/319486852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f28102da0 0x7f7f28198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 >> 172.21.15.78:0/319486852 conn(0x7f7f280fc7a0 msgr2=0x7f7f280fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 shutdown_connections 2024-09-16T01:10:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:09.549+0000 7f7f3064f640 1 -- 172.21.15.78:0/319486852 wait complete. 2024-09-16T01:10:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:09 smithi148 ceph-mon[35030]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:10.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:10 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/319486852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:10.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:10 smithi078 ceph-mon[30937]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:10.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:10:10.526 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:10 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/319486852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:10 smithi193 ceph-mon[35701]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:10 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/319486852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:10 smithi148 ceph-mon[35030]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:11.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 -- 172.21.15.78:0/682678812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a1410c170 msgr2=0x7f2a1410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 --2- 172.21.15.78:0/682678812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a1410c170 0x7f2a1410e610 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009ca0 tx=0x7f2a0c02a950 comp rx=0 tx=0).stop 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 -- 172.21.15.78:0/682678812 shutdown_connections 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 --2- 172.21.15.78:0/682678812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a1410c170 0x7f2a1410e610 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 --2- 172.21.15.78:0/682678812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a14103660 0x7f2a1410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 --2- 172.21.15.78:0/682678812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 0x7f2a14103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 -- 172.21.15.78:0/682678812 >> 172.21.15.78:0/682678812 conn(0x7f2a140fc7a0 msgr2=0x7f2a140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 -- 172.21.15.78:0/682678812 shutdown_connections 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.006+0000 7f2a19055640 1 -- 172.21.15.78:0/682678812 wait complete. 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.007+0000 7f2a19055640 1 Processor -- start 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.007+0000 7f2a19055640 1 -- start start 2024-09-16T01:10:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.007+0000 7f2a19055640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 0x7f2a14198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.007+0000 7f2a19055640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 0x7f2a1419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14111f70 con 0x7f2a14103660 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14111df0 con 0x7f2a1410c170 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2a141120f0 con 0x7f2a14102d40 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42092/0 (socket says 172.21.15.78:42092) 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 -- 172.21.15.78:0/779327580 learned_addr learned my addr 172.21.15.78:0/779327580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1359e640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 0x7f2a1419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a12d9d640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 0x7f2a14198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 -- 172.21.15.78:0/779327580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 msgr2=0x7f2a14198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:11.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 0x7f2a14198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 -- 172.21.15.78:0/779327580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 msgr2=0x7f2a1419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 0x7f2a1419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 -- 172.21.15.78:0/779327580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1419d3c0 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1359e640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 0x7f2a1419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a1259c640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800d740 tx=0x7f2a0800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08017070 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a0800de50 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800f620 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1419d6b0 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.008+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a141a4f90 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.009+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a14103120 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.010+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2a08009c30 con 0x7f2a14103660 2024-09-16T01:10:11.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.011+0000 7f29fbfff640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 0x7f29fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:11.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.011+0000 7f2a12d9d640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 0x7f29fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:11.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.012+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2a080cfa20 con 0x7f2a14103660 2024-09-16T01:10:11.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.012+0000 7f2a12d9d640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 0x7f29fc0a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a04005fd0 tx=0x7f2a04005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:11.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.016+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2a0809c280 con 0x7f2a14103660 2024-09-16T01:10:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.185+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a141a51c0 con 0x7f2a14103660 2024-09-16T01:10:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.187+0000 7f29fbfff640 1 -- 172.21.15.78:0/779327580 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24645 (secure 0 0 0) 0x7f2a0809bb80 con 0x7f2a14103660 2024-09-16T01:10:11.188 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:11.188 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:09:39.105086+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:10:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 msgr2=0x7f29fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 0x7f29fc0a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a04005fd0 tx=0x7f2a04005850 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 msgr2=0x7f2a14198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800d740 tx=0x7f2a0800dc00 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 shutdown_connections 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f29fc0a32d0 0x7f29fc0a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a1410c170 0x7f2a1419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a14103660 0x7f2a14198950 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 --2- 172.21.15.78:0/779327580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a14102d40 0x7f2a14198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 >> 172.21.15.78:0/779327580 conn(0x7f2a140fc7a0 msgr2=0x7f2a140fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 shutdown_connections 2024-09-16T01:10:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:11.189+0000 7f2a19055640 1 -- 172.21.15.78:0/779327580 wait complete. 2024-09-16T01:10:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:11 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/779327580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:11 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/779327580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:11 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/779327580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:11.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:12.177 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:12.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:12 smithi078 ceph-mon[30937]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:12.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:12 smithi193 ceph-mon[35701]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:12.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:12 smithi148 ceph-mon[35030]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 -- 172.21.15.78:0/3116393015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 msgr2=0x7f386c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 --2- 172.21.15.78:0/3116393015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c103190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f385c009cd0 tx=0x7f385c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 -- 172.21.15.78:0/3116393015 shutdown_connections 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 --2- 172.21.15.78:0/3116393015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c1082e0 0x7f386c10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 --2- 172.21.15.78:0/3116393015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c103760 0x7f386c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 --2- 172.21.15.78:0/3116393015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c103190 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.709+0000 7f3873831640 1 -- 172.21.15.78:0/3116393015 >> 172.21.15.78:0/3116393015 conn(0x7f386c0fc7d0 msgr2=0x7f386c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- 172.21.15.78:0/3116393015 shutdown_connections 2024-09-16T01:10:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- 172.21.15.78:0/3116393015 wait complete. 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 Processor -- start 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- start start 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c103760 0x7f386c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c1082e0 0x7f386c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1120b0 con 0x7f386c102db0 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f386c111f30 con 0x7f386c1082e0 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f3873831640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f386c112230 con 0x7f386c103760 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f38715cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f38715cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42120/0 (socket says 172.21.15.78:42120) 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.710+0000 7f38715cd640 1 -- 172.21.15.78:0/1399717151 learned_addr learned my addr 172.21.15.78:0/1399717151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f3871dce640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c1082e0 0x7f386c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f3870dcc640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c103760 0x7f386c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 -- 172.21.15.78:0/1399717151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c103760 msgr2=0x7f386c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c103760 0x7f386c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 -- 172.21.15.78:0/1399717151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c1082e0 msgr2=0x7f386c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c1082e0 0x7f386c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 -- 172.21.15.78:0/1399717151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386c19d540 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f38715cd640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f385c02ae00 tx=0x7f385c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c004240 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f385c031d50 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386c19d7d0 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c03a400 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.711+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386c1a5110 con 0x7f386c102db0 2024-09-16T01:10:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.712+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f386c103190 con 0x7f386c102db0 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.713+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f385c0306c0 con 0x7f386c102db0 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.714+0000 7f386a7fc640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 0x7f38540a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.714+0000 7f3870dcc640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 0x7f38540a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.714+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f385c02e030 con 0x7f386c102db0 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.714+0000 7f3870dcc640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 0x7f38540a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f386c0fee20 tx=0x7f3860005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.717+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f385c0bdd00 con 0x7f386c102db0 2024-09-16T01:10:12.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.922+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f386c10a2c0 con 0x7f386c102db0 2024-09-16T01:10:12.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.922+0000 7f386a7fc640 1 -- 172.21.15.78:0/1399717151 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7f385c0bd600 con 0x7f386c102db0 2024-09-16T01:10:12.924 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:12.924 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:10:12.924 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:12.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.924+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 msgr2=0x7f38540a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:12.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.924+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 0x7f38540a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f386c0fee20 tx=0x7f3860005c60 comp rx=0 tx=0).stop 2024-09-16T01:10:12.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.924+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 msgr2=0x7f386c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.924+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f385c02ae00 tx=0x7f385c003ba0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 shutdown_connections 2024-09-16T01:10:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f38540a32d0 0x7f38540a5790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f386c1082e0 0x7f386c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c103760 0x7f386c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 --2- 172.21.15.78:0/1399717151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f386c102db0 0x7f386c198590 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 >> 172.21.15.78:0/1399717151 conn(0x7f386c0fc7d0 msgr2=0x7f386c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 shutdown_connections 2024-09-16T01:10:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:12.925+0000 7f3873831640 1 -- 172.21.15.78:0/1399717151 wait complete. 2024-09-16T01:10:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:13 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1399717151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:13 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1399717151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:13.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:13 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1399717151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:13.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T01:08:29.385380+0000', 'modified': '2024-09-16T01:09:07.935774+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24485, 'mds_1': 24460, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1399158659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1399158659}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1399158659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/1850679548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 1850679548}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 1850679548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/1637418104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1637418104}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1637418104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.148:6837/330388800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 330388800}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 330388800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.193:6839/1136630649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 1136630649}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 1136630649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6839/3106047695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 3106047695}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 3106047695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24455, 24460, 24485]} 2024-09-16T01:10:13.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T01:10:13.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-16T01:10:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:14.416 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:14 smithi078 ceph-mon[30937]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:14.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 -- 172.21.15.78:0/1097053863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101240 msgr2=0x7f9f8c101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 --2- 172.21.15.78:0/1097053863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101240 0x7f9f8c101680 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009900 tx=0x7f9f8802a410 comp rx=0 tx=0).stop 2024-09-16T01:10:14.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 -- 172.21.15.78:0/1097053863 shutdown_connections 2024-09-16T01:10:14.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 --2- 172.21.15.78:0/1097053863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c10d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 --2- 172.21.15.78:0/1097053863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101240 0x7f9f8c101680 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 --2- 172.21.15.78:0/1097053863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 0x7f9f8c105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.495+0000 7f9f946e7640 1 -- 172.21.15.78:0/1097053863 >> 172.21.15.78:0/1097053863 conn(0x7f9f8c077210 msgr2=0x7f9f8c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- 172.21.15.78:0/1097053863 shutdown_connections 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- 172.21.15.78:0/1097053863 wait complete. 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 Processor -- start 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- start start 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101240 0x7f9f8c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 0x7f9f8c1a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c110f70 con 0x7f9f8c101bc0 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c110df0 con 0x7f9f8c105a90 2024-09-16T01:10:14.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f946e7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c1110f0 con 0x7f9f8c101240 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34808/0 (socket says 172.21.15.78:34808) 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.496+0000 7f9f91c82640 1 -- 172.21.15.78:0/179935311 learned_addr learned my addr 172.21.15.78:0/179935311 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f92483640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101240 0x7f9f8c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f92c84640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 0x7f9f8c1a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 -- 172.21.15.78:0/179935311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101240 msgr2=0x7f9f8c199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101240 0x7f9f8c199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 -- 172.21.15.78:0/179935311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 msgr2=0x7f9f8c1a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 0x7f9f8c1a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 -- 172.21.15.78:0/179935311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c1a4900 con 0x7f9f8c101bc0 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f91c82640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006010 tx=0x7f9f88003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f7f7fe640 1 -- 172.21.15.78:0/179935311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8803f070 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c1a4ad0 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f7f7fe640 1 -- 172.21.15.78:0/179935311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f88031e30 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f7f7fe640 1 -- 172.21.15.78:0/179935311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f88035030 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.497+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c1a4f30 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.499+0000 7f9f7f7fe640 1 -- 172.21.15.78:0/179935311 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9f88042070 con 0x7f9f8c101bc0 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 Processor -- start 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 -- start start 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bd88c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd9a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563e6bcef090 con 0x563e6bd891d0 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6bc96a70 con 0x563e6bd97fc0 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563e6bceb050 con 0x563e6bd888b0 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f7f7fe640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 0x7f9f780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f92483640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f92483640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34812/0 (socket says 172.21.15.78:34812) 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f92483640 1 -- 172.21.15.78:0/1952165733 learned_addr learned my addr 172.21.15.78:0/1952165733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:14.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f92c84640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 0x7f9f780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f7f7fe640 1 -- 172.21.15.78:0/179935311 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f880f0800 con 0x7f9f8c101bc0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f91c82640 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd9a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.500+0000 7f9f92c84640 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bd88c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 -- 172.21.15.78:0/1952165733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 msgr2=0x563e6bd88c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bd88c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 -- 172.21.15.78:0/1952165733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 msgr2=0x563e6bd9a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd9a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 -- 172.21.15.78:0/1952165733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e6bca3130 con 0x563e6bd891d0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92c84640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 0x7f9f780a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005c40 tx=0x7f9f8400a3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f92483640 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000cbe0 tx=0x7f9f8000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53dd93c0a0e4d09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f7cff9640 1 -- 172.21.15.78:0/1952165733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80005220 con 0x563e6bd891d0 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f7cff9640 1 -- 172.21.15.78:0/1952165733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f80004c80 con 0x563e6bd891d0 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.501+0000 7f9f7cff9640 1 -- 172.21.15.78:0/1952165733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f800108d0 con 0x563e6bd891d0 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- 172.21.15.78:0/1952165733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 msgr2=0x563e6bd97a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000cbe0 tx=0x7f9f8000e720 comp rx=0 tx=0).stop 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- 172.21.15.78:0/1952165733 shutdown_connections 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd9a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd97a80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/1952165733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bd88c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- 172.21.15.78:0/1952165733 >> 172.21.15.78:0/1952165733 conn(0x563e6bd84f60 msgr2=0x563e6bd85340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- 172.21.15.78:0/1952165733 shutdown_connections 2024-09-16T01:10:14.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- 172.21.15.78:0/1952165733 wait complete. 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 -- start start 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.502+0000 7f9f96ac0740 1 Processor -- start 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bd97820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bd950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563e6bd9ab90 con 0x563e6bd891d0 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6bca9f20 con 0x563e6bd888b0 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563e6bd9ad10 con 0x563e6bd97fc0 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 10 client.0 _ll_register_callbacks cb 0x563e6bc73030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:14.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34816/0 (socket says 172.21.15.78:34816) 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 -- 172.21.15.78:0/178442368 learned_addr learned my addr 172.21.15.78:0/178442368 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f92483640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bd97820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f92c84640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bd950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 -- 172.21.15.78:0/178442368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 msgr2=0x563e6bd950b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bd950b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 -- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 msgr2=0x563e6bd97820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bd97820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 -- 172.21.15.78:0/178442368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e6ba602b0 con 0x563e6bd891d0 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f92483640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bd97820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:14.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f91c82640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b8dc0 tx=0x7f9f88034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bde351bc37182c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:14.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f880d5060 con 0x563e6bd891d0 2024-09-16T01:10:14.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f880b8fd0 con 0x563e6bd891d0 2024-09-16T01:10:14.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f80004c80 con 0x563e6bd891d0 2024-09-16T01:10:14.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.503+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9f88049040 con 0x563e6bd891d0 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 10 client.15048 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e6bda64c0 con 0x563e6bd891d0 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f80004c80 con 0x563e6bd891d0 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f9f8803b5f0 con 0x563e6bd891d0 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 10 client.15048 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 10 client.15048 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 20 client.15048 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6bda8b20 0x563e6bda9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 4 client.15048 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e6bcfc600 con 0x563e6bda8b20 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.504+0000 7f9f92483640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6bda8b20 0x563e6bda9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f92483640 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6bda8b20 0x563e6bda9f90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3de7c271fd7f4dc0 server_cookie=3fa4deb743a10c20 in_seq=0 out_seq=0 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f75ffb640 10 client.15048 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f75ffb640 1 -- 172.21.15.78:0/178442368 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e6bcfc600 con 0x563e6bda8b20 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f75ffb640 10 client.15048 handle_command_reply: tid=0 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f96ac0740 1 client.15048 shutdown 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.505+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 msgr2=0x563e6bd94b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b8dc0 tx=0x7f9f88034000 comp rx=0 tx=0).stop 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 shutdown_connections 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6bda8b20 0x563e6bda9f90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bd950b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd94b70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/178442368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bd97820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f75ffb640 0 client.15048 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 >> 172.21.15.78:0/178442368 conn(0x563e6bd84f60 msgr2=0x563e6bd8acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 shutdown_connections 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 1 -- 172.21.15.78:0/178442368 wait complete. 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.506+0000 7f9f96ac0740 20 client.15048 trim_cache size 0 max 16384 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.523+0000 7f9f96ac0740 1 Processor -- start 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.523+0000 7f9f96ac0740 1 -- start start 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bda9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd93980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563e6bda82d0 con 0x563e6bd891d0 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6bda89a0 con 0x563e6bd97fc0 2024-09-16T01:10:14.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563e6bda9070 con 0x563e6bd888b0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34824/0 (socket says 172.21.15.78:34824) 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 -- 172.21.15.78:0/3027590026 learned_addr learned my addr 172.21.15.78:0/3027590026 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f92483640 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bda9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 -- 172.21.15.78:0/3027590026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 msgr2=0x563e6bda9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bda9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 -- 172.21.15.78:0/3027590026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 msgr2=0x563e6bd93980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd93980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 -- 172.21.15.78:0/3027590026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e6bda47e0 con 0x563e6bd891d0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f92483640 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bda9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9f880bc660 tx=0x7f9f880bae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3009f660fc310e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f77fff640 1 -- 172.21.15.78:0/3027590026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f880bd060 con 0x563e6bd891d0 2024-09-16T01:10:14.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f77fff640 1 -- 172.21.15.78:0/3027590026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f880c0d10 con 0x563e6bd891d0 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.524+0000 7f9f77fff640 1 -- 172.21.15.78:0/3027590026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f880c5140 con 0x563e6bd891d0 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3027590026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 msgr2=0x563e6bd93440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9f880bc660 tx=0x7f9f880bae70 comp rx=0 tx=0).stop 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3027590026 shutdown_connections 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd97fc0 0x563e6bd93980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bd93440 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3027590026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd888b0 0x563e6bda9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3027590026 >> 172.21.15.78:0/3027590026 conn(0x563e6bd84f60 msgr2=0x563e6bd7fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3027590026 shutdown_connections 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3027590026 wait complete. 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 -- start start 2024-09-16T01:10:14.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.525+0000 7f9f96ac0740 1 Processor -- start 2024-09-16T01:10:14.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bdb2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bdab750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563e6bda9e10 con 0x563e6bd891d0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6bda9740 con 0x563e6bd888b0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563e6bdaa4e0 con 0x563e6bd97fc0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 10 client.0 _ll_register_callbacks cb 0x7f9f88005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34832/0 (socket says 172.21.15.78:34832) 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 -- 172.21.15.78:0/3251603360 learned_addr learned my addr 172.21.15.78:0/3251603360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f92483640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bdb2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f92c84640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bdab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 -- 172.21.15.78:0/3251603360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 msgr2=0x563e6bdab750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bdab750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 -- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 msgr2=0x563e6bdb2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bdb2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 -- 172.21.15.78:0/3251603360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e6bdabf60 con 0x563e6bd891d0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f92483640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bdb2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f91c82640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9f880bc8f0 tx=0x7f9f880edd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f83d354d177be2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f880bd060 con 0x563e6bd891d0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f880b8fd0 con 0x563e6bd891d0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.526+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f880c0d10 con 0x563e6bd891d0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9f880bcbc0 con 0x563e6bd891d0 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 10 client.15054 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:14.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e6bdabf60 con 0x563e6bd891d0 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f880c0d10 con 0x563e6bd891d0 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f9f880bbeb0 con 0x563e6bd891d0 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 10 client.15054 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 10 client.15054 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 20 client.15054 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6be17210 0x563e6be15f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 4 client.15054 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "status"}) -- 0x563e6bcf6b10 con 0x563e6be17210 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.527+0000 7f9f92483640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6be17210 0x563e6be15f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f75ffb640 10 client.15054 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f92483640 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6be17210 0x563e6be15f20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f14be3e595ac42a server_cookie=e5f3b1662c3e866 in_seq=0 out_seq=0 2024-09-16T01:10:14.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f75ffb640 1 -- 172.21.15.78:0/3251603360 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563e6bcf6b10 con 0x563e6be17210 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f75ffb640 10 client.15054 handle_command_reply: tid=0 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 client.15054 shutdown 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 msgr2=0x563e6bdb2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9f880bc8f0 tx=0x7f9f880edd90 comp rx=0 tx=0).stop 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 shutdown_connections 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x563e6be17210 0x563e6be15f20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563e6bd97fc0 0x563e6bdab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563e6bd891d0 0x563e6bdb2960 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 --2- 172.21.15.78:0/3251603360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6bd888b0 0x563e6bdb2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 >> 172.21.15.78:0/3251603360 conn(0x563e6bd84f60 msgr2=0x563e6bd933c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f75ffb640 0 client.15054 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 shutdown_connections 2024-09-16T01:10:14.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 1 -- 172.21.15.78:0/3251603360 wait complete. 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.528+0000 7f9f96ac0740 20 client.15054 trim_cache size 0 max 16384 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24455, 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 69.434477146000006, 2024-09-16T01:10:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-09-16T01:10:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 127.84149175500001 2024-09-16T01:10:14.551 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 msgr2=0x7f9f780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 0x7f9f780a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005c40 tx=0x7f9f8400a3c0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 msgr2=0x7f9f8c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f88006010 tx=0x7f9f88003c80 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 shutdown_connections 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f9f780a32d0 0x7f9f780a5790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f8c105a90 0x7f9f8c1a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f8c101bc0 0x7f9f8c199880 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 --2- 172.21.15.78:0/179935311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f8c101240 0x7f9f8c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 >> 172.21.15.78:0/179935311 conn(0x7f9f8c077210 msgr2=0x7f9f8c103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 shutdown_connections 2024-09-16T01:10:14.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:14.529+0000 7f9f946e7640 1 -- 172.21.15.78:0/179935311 wait complete. 2024-09-16T01:10:14.553 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:14 smithi193 ceph-mon[35701]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:14.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:14 smithi148 ceph-mon[35030]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:15.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T01:10:15.619 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:16.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 -- 172.21.15.78:0/3676179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 msgr2=0x7f1154106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 --2- 172.21.15.78:0/3676179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f1154106e60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1148009930 tx=0x7f114802a420 comp rx=0 tx=0).stop 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 -- 172.21.15.78:0/3676179 shutdown_connections 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 --2- 172.21.15.78:0/3676179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 0x7f115410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 --2- 172.21.15.78:0/3676179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 --2- 172.21.15.78:0/3676179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f1154106e60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 -- 172.21.15.78:0/3676179 >> 172.21.15.78:0/3676179 conn(0x7f11540fc7a0 msgr2=0x7f11540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 -- 172.21.15.78:0/3676179 shutdown_connections 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.164+0000 7f115acf9640 1 -- 172.21.15.78:0/3676179 wait complete. 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 Processor -- start 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 -- start start 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 0x7f1154199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1154110e80 con 0x7f1154106a80 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1154110d00 con 0x7f1154101420 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f115acf9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1154111000 con 0x7f1154100a80 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f1159296640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f1159296640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34854/0 (socket says 172.21.15.78:34854) 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f1159296640 1 -- 172.21.15.78:0/4155719088 learned_addr learned my addr 172.21.15.78:0/4155719088 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.165+0000 7f1153fff640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 0x7f1154199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1158a95640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 -- 172.21.15.78:0/4155719088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 msgr2=0x7f1154199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 -- 172.21.15.78:0/4155719088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 msgr2=0x7f1154199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 0x7f1154199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 -- 172.21.15.78:0/4155719088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11541a48d0 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1158a95640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1159296640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f114c00eaa0 tx=0x7f114c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1151ffb640 1 -- 172.21.15.78:0/4155719088 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c00cdb0 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1151ffb640 1 -- 172.21.15.78:0/4155719088 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f114c005020 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11541a4aa0 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f1151ffb640 1 -- 172.21.15.78:0/4155719088 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c0116b0 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.166+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11541a4f60 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.168+0000 7f1151ffb640 1 -- 172.21.15.78:0/4155719088 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f114c005560 con 0x7f1154106a80 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1151ffb640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 0x7f11440a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 Processor -- start 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 -- start start 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1158a95640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 0x7f11440a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1151ffb640 1 -- 172.21.15.78:0/4155719088 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f114c0d0010 con 0x7f1154106a80 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 0x55c4a9d15cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d161f0 0x55c4a9d24a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9c7c090 con 0x55c4a9d24fc0 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9c23a70 con 0x55c4a9d161f0 2024-09-16T01:10:16.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f115d0d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9c78050 con 0x55c4a9d158d0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1159296640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1159296640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34864/0 (socket says 172.21.15.78:34864) 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1159296640 1 -- 172.21.15.78:0/3364328171 learned_addr learned my addr 172.21.15.78:0/3364328171 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1153fff640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d161f0 0x55c4a9d24a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.169+0000 7f1158a95640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 0x55c4a9d15cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1158a95640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 0x7f11440a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1148000c00 tx=0x7f1148032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 -- 172.21.15.78:0/3364328171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 msgr2=0x55c4a9d15cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 0x55c4a9d15cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 -- 172.21.15.78:0/3364328171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d161f0 msgr2=0x55c4a9d24a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d161f0 0x55c4a9d24a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 -- 172.21.15.78:0/3364328171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a9c30130 con 0x55c4a9d24fc0 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1158a95640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 0x55c4a9d15cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:16.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f1159296640 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f114c0d02f0 tx=0x7f114c09bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b20f236337844bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f113effd640 1 -- 172.21.15.78:0/3364328171 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c01f030 con 0x55c4a9d24fc0 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f113effd640 1 -- 172.21.15.78:0/3364328171 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f114c0cd8b0 con 0x55c4a9d24fc0 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.170+0000 7f113effd640 1 -- 172.21.15.78:0/3364328171 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c0a0120 con 0x55c4a9d24fc0 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- 172.21.15.78:0/3364328171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 msgr2=0x55c4a9d273b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f114c0d02f0 tx=0x7f114c09bc00 comp rx=0 tx=0).stop 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- 172.21.15.78:0/3364328171 shutdown_connections 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d273b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d161f0 0x55c4a9d24a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 --2- 172.21.15.78:0/3364328171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d158d0 0x55c4a9d15cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- 172.21.15.78:0/3364328171 >> 172.21.15.78:0/3364328171 conn(0x55c4a9d11f60 msgr2=0x55c4a9d12340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- 172.21.15.78:0/3364328171 shutdown_connections 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- 172.21.15.78:0/3364328171 wait complete. 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 -- start start 2024-09-16T01:10:16.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.171+0000 7f115d0d2740 1 Processor -- start 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d24840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d220d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d27bf0 con 0x55c4a9d161f0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9c36f20 con 0x55c4a9d158d0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d27d70 con 0x55c4a9d24fc0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 10 client.0 _ll_register_callbacks cb 0x55c4a9c00030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34866/0 (socket says 172.21.15.78:34866) 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 -- 172.21.15.78:0/610319177 learned_addr learned my addr 172.21.15.78:0/610319177 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1158a95640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d24840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1159296640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 -- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 msgr2=0x55c4a9d220d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d220d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 -- 172.21.15.78:0/610319177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 msgr2=0x55c4a9d24840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d24840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 -- 172.21.15.78:0/610319177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a99ed2b0 con 0x55c4a9d161f0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1159296640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1158a95640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d24840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1153fff640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f114000beb0 tx=0x7f1140012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3bca814beb9d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1133fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140005220 con 0x55c4a9d161f0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114c0cd8b0 con 0x55c4a9d161f0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f11400053c0 con 0x55c4a9d161f0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1140025050 con 0x55c4a9d161f0 2024-09-16T01:10:16.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.172+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4a9d33540 con 0x55c4a9d161f0 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f114c0cd8b0 con 0x55c4a9d161f0 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f1140019030 con 0x55c4a9d161f0 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 10 client.15066 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 20 client.15066 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9d37070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 4 client.15066 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4a9c89600 con 0x55c4a9d35c00 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.173+0000 7f1158a95640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9d37070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f1133fff640 10 client.15066 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f1158a95640 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9d37070 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1937138f78ee717f server_cookie=727b4ca12fdc4736 in_seq=0 out_seq=0 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f1133fff640 1 -- 172.21.15.78:0/610319177 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4a9c89600 con 0x55c4a9d35c00 2024-09-16T01:10:16.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f1133fff640 10 client.15066 handle_command_reply: tid=0 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 client.15066 shutdown 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 msgr2=0x55c4a9d21b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f114000beb0 tx=0x7f1140012820 comp rx=0 tx=0).stop 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 shutdown_connections 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9d37070 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f1133fff640 0 client.15066 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d21b90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.174+0000 7f115d0d2740 1 --2- 172.21.15.78:0/610319177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d24840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.175+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 >> 172.21.15.78:0/610319177 conn(0x55c4a9d11f60 msgr2=0x55c4a9d17cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.175+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 shutdown_connections 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.175+0000 7f115d0d2740 1 -- 172.21.15.78:0/610319177 wait complete. 2024-09-16T01:10:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.175+0000 7f115d0d2740 20 client.15066 trim_cache size 0 max 16384 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 Processor -- start 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 -- start start 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d37070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d1edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d353b0 con 0x55c4a9d161f0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d35a80 con 0x55c4a9d158d0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f115d0d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d36150 con 0x55c4a9d24fc0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f1153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f1153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34872/0 (socket says 172.21.15.78:34872) 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.192+0000 7f1153fff640 1 -- 172.21.15.78:0/1242200354 learned_addr learned my addr 172.21.15.78:0/1242200354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1159296640 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d1edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 -- 172.21.15.78:0/1242200354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 msgr2=0x55c4a9d1edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1158a95640 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d37070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d1edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 -- 172.21.15.78:0/1242200354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 msgr2=0x55c4a9d37070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d37070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 -- 172.21.15.78:0/1242200354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a9d130f0 con 0x55c4a9d161f0 2024-09-16T01:10:16.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f1153fff640 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1140010bc0 tx=0x7f1140005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570c1165566557ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f113e7fc640 1 -- 172.21.15.78:0/1242200354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114002d070 con 0x55c4a9d161f0 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f113e7fc640 1 -- 172.21.15.78:0/1242200354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400107d0 con 0x55c4a9d161f0 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.193+0000 7f113e7fc640 1 -- 172.21.15.78:0/1242200354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140029690 con 0x55c4a9d161f0 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- 172.21.15.78:0/1242200354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 msgr2=0x55c4a9d1d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1140010bc0 tx=0x7f1140005670 comp rx=0 tx=0).stop 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- 172.21.15.78:0/1242200354 shutdown_connections 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d1edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d161f0 0x55c4a9d1d130 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 --2- 172.21.15.78:0/1242200354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d158d0 0x55c4a9d37070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- 172.21.15.78:0/1242200354 >> 172.21.15.78:0/1242200354 conn(0x55c4a9d11f60 msgr2=0x55c4a9d1ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- 172.21.15.78:0/1242200354 shutdown_connections 2024-09-16T01:10:16.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- 172.21.15.78:0/1242200354 wait complete. 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 -- start start 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.194+0000 7f115d0d2740 1 Processor -- start 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d161f0 0x55c4a9d38d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d39290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d36ef0 con 0x55c4a9d158d0 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d36820 con 0x55c4a9d24fc0 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a9d375c0 con 0x55c4a9d161f0 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f115d0d2740 10 client.0 _ll_register_callbacks cb 0x7f114000bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34882/0 (socket says 172.21.15.78:34882) 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 -- 172.21.15.78:0/2541821758 learned_addr learned my addr 172.21.15.78:0/2541821758 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1153fff640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d161f0 0x55c4a9d38d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1159296640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d39290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 -- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d161f0 msgr2=0x55c4a9d38d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d161f0 0x55c4a9d38d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 -- 172.21.15.78:0/2541821758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d24fc0 msgr2=0x55c4a9d39290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d39290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 -- 172.21.15.78:0/2541821758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a9d39ac0 con 0x55c4a9d158d0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1158a95640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1148032fd0 tx=0x7f1148009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65439882dc968d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.195+0000 7f1133fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148035050 con 0x55c4a9d158d0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1148005020 con 0x55c4a9d158d0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11400107d0 con 0x55c4a9d158d0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f114805c050 con 0x55c4a9d158d0 2024-09-16T01:10:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 10 client.15072 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4a9da0370 con 0x55c4a9d158d0 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11400107d0 con 0x55c4a9d158d0 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f11480035f0 con 0x55c4a9d158d0 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 10 client.15072 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 10 client.15072 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 20 client.15072 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9da3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 4 client.15072 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "status"}) -- 0x55c4a9c83b10 con 0x55c4a9d35c00 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.196+0000 7f1153fff640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9da3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f1153fff640 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9da3c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc43463e9867fda3 server_cookie=516310efb84ecdcf in_seq=0 out_seq=0 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f1133fff640 10 client.15072 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f1133fff640 1 -- 172.21.15.78:0/2541821758 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c4a9c83b10 con 0x55c4a9d35c00 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f1133fff640 10 client.15072 handle_command_reply: tid=0 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 client.15072 shutdown 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 msgr2=0x55c4a9d3fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1148032fd0 tx=0x7f1148009260 comp rx=0 tx=0).stop 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 shutdown_connections 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55c4a9d35c00 0x55c4a9da3c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4a9d24fc0 0x55c4a9d39290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a9d161f0 0x55c4a9d38d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 --2- 172.21.15.78:0/2541821758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4a9d158d0 0x55c4a9d3fa20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f1133fff640 0 client.15072 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 >> 172.21.15.78:0/2541821758 conn(0x55c4a9d11f60 msgr2=0x55c4a9d1d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 shutdown_connections 2024-09-16T01:10:16.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 1 -- 172.21.15.78:0/2541821758 wait complete. 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.197+0000 7f115d0d2740 20 client.15072 trim_cache size 0 max 16384 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24460, 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:16.208 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 72.112385051999993, 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 126.98174711199999 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:16.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 msgr2=0x7f11440a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 0x7f11440a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1148000c00 tx=0x7f1148032c70 comp rx=0 tx=0).stop 2024-09-16T01:10:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 msgr2=0x7f11541a4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f114c00eaa0 tx=0x7f114c00ef60 comp rx=0 tx=0).stop 2024-09-16T01:10:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 shutdown_connections 2024-09-16T01:10:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f11440a32d0 0x7f11440a5790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1154106a80 0x7f11541a4390 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1154101420 0x7f1154199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 --2- 172.21.15.78:0/4155719088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1154100a80 0x7f1154199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 >> 172.21.15.78:0/4155719088 conn(0x7f11540fc7a0 msgr2=0x7f11540fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 shutdown_connections 2024-09-16T01:10:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:16.198+0000 7f115acf9640 1 -- 172.21.15.78:0/4155719088 wait complete. 2024-09-16T01:10:17.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T01:10:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:17 smithi148 ceph-mon[35030]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:17.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:17 smithi078 ceph-mon[30937]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:17 smithi193 ceph-mon[35701]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 -- 172.21.15.78:0/2837747978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc106f10 msgr2=0x7f12cc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 --2- 172.21.15.78:0/2837747978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc106f10 0x7f12cc10ea90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009ca0 tx=0x7f12c402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 -- 172.21.15.78:0/2837747978 shutdown_connections 2024-09-16T01:10:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 --2- 172.21.15.78:0/2837747978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc106f10 0x7f12cc10ea90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 --2- 172.21.15.78:0/2837747978 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 --2- 172.21.15.78:0/2837747978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12cc101de0 0x7f12cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.950+0000 7f12d1830640 1 -- 172.21.15.78:0/2837747978 >> 172.21.15.78:0/2837747978 conn(0x7f12cc0fd8c0 msgr2=0x7f12cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:17.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- 172.21.15.78:0/2837747978 shutdown_connections 2024-09-16T01:10:17.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- 172.21.15.78:0/2837747978 wait complete. 2024-09-16T01:10:17.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 Processor -- start 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- start start 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12cc106f10 0x7f12cc074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc112010 con 0x7f12cc101de0 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc111e90 con 0x7f12cc102790 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12d1830640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc112190 con 0x7f12cc106f10 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34896/0 (socket says 172.21.15.78:34896) 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.951+0000 7f12caffd640 1 -- 172.21.15.78:0/2009522455 learned_addr learned my addr 172.21.15.78:0/2009522455 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 -- 172.21.15.78:0/2009522455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12cc106f10 msgr2=0x7f12cc074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12cc106f10 0x7f12cc074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 -- 172.21.15.78:0/2009522455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 msgr2=0x7f12cc074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 -- 172.21.15.78:0/2009522455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc1a6a30 con 0x7f12cc101de0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12caffd640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b4a0 tx=0x7f12bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12d082e640 1 -- 172.21.15.78:0/2009522455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc014070 con 0x7f12cc101de0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12cc1a6d20 con 0x7f12cc101de0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12d082e640 1 -- 172.21.15.78:0/2009522455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12bc010af0 con 0x7f12cc101de0 2024-09-16T01:10:17.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12d082e640 1 -- 172.21.15.78:0/2009522455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc017d10 con 0x7f12cc101de0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.952+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12cc1a7200 con 0x7f12cc101de0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.954+0000 7f12d082e640 1 -- 172.21.15.78:0/2009522455 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f12bc017070 con 0x7f12cc101de0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 Processor -- start 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 -- start start 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 0x555896ae6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896af3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555896a4d090 con 0x555896ae72f0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5558969f4a70 con 0x555896ae69d0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d3c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555896a49050 con 0x555896af1bb0 2024-09-16T01:10:17.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d082e640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 0x7f12b80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34912/0 (socket says 172.21.15.78:34912) 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 -- 172.21.15.78:0/1064528185 learned_addr learned my addr 172.21.15.78:0/1064528185 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 0x7f12b80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 0x555896ae6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12c3fff640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896af3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12d082e640 1 -- 172.21.15.78:0/2009522455 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12bc0cf110 con 0x7f12cc101de0 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 -- 172.21.15.78:0/1064528185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 msgr2=0x555896af3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896af3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 -- 172.21.15.78:0/1064528185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 msgr2=0x555896ae6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 0x555896ae6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12caffd640 1 -- 172.21.15.78:0/1064528185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555896a01130 con 0x555896ae72f0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 0x555896ae6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.955+0000 7f12c3fff640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896af3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.956+0000 7f12caffd640 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0cc7e0 tx=0x7f12bc09b200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de63dad561dbf6d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.956+0000 7f12c1ffb640 1 -- 172.21.15.78:0/1064528185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc01e070 con 0x555896ae72f0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.956+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 0x7f12b80a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.956+0000 7f12c1ffb640 1 -- 172.21.15.78:0/1064528185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12bc09ba80 con 0x555896ae72f0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.956+0000 7f12c1ffb640 1 -- 172.21.15.78:0/1064528185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0cc9b0 con 0x555896ae72f0 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- 172.21.15.78:0/1064528185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 msgr2=0x555896af1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0cc7e0 tx=0x7f12bc09b200 comp rx=0 tx=0).stop 2024-09-16T01:10:17.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- 172.21.15.78:0/1064528185 shutdown_connections 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896af3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae72f0 0x555896af1670 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1064528185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae69d0 0x555896ae6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- 172.21.15.78:0/1064528185 >> 172.21.15.78:0/1064528185 conn(0x555896ae2f60 msgr2=0x555896ae3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- 172.21.15.78:0/1064528185 shutdown_connections 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- 172.21.15.78:0/1064528185 wait complete. 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- start start 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 Processor -- start 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896aee820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896aeed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555896af4780 con 0x555896ae69d0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555896a07f20 con 0x555896ae72f0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.957+0000 7f12d3c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555896af4900 con 0x555896af1bb0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12d3c09740 10 client.0 _ll_register_callbacks cb 0x5558969d1030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34922/0 (socket says 172.21.15.78:34922) 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 -- 172.21.15.78:0/2031096428 learned_addr learned my addr 172.21.15.78:0/2031096428 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896aeed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896aee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 -- 172.21.15.78:0/2031096428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 msgr2=0x555896aeed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896aeed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 -- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 msgr2=0x555896aee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896aee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 -- 172.21.15.78:0/2031096428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558967be2b0 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12caffd640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b470 tx=0x7f12bc009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc0a99f4682931 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc01e070 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12bc010ca0 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12bc09ba80 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f12bc0a3ff0 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12d3c09740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555896b00100 con 0x555896ae69d0 2024-09-16T01:10:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.958+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12bc09ba80 con 0x555896ae69d0 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f12bc0d10d0 con 0x555896ae69d0 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 20 client.15084 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b02290 0x555896b03700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 4 client.15084 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555896a5a600 con 0x555896b02290 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.959+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b02290 0x555896b03700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12aaffd640 10 client.15084 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b02290 0x555896b03700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=478c27bf66dbcbd8 server_cookie=6da146e1b7e59a61 in_seq=0 out_seq=0 2024-09-16T01:10:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12aaffd640 1 -- 172.21.15.78:0/2031096428 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555896a5a600 con 0x555896b02290 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12aaffd640 10 client.15084 handle_command_reply: tid=0 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 client.15084 shutdown 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 msgr2=0x555896af14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b470 tx=0x7f12bc009390 comp rx=0 tx=0).stop 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 shutdown_connections 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b02290 0x555896b03700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896af1bb0 0x555896aeed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896aee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2031096428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896af14d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.960+0000 7f12aaffd640 0 client.15084 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.961+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 >> 172.21.15.78:0/2031096428 conn(0x555896ae2f60 msgr2=0x555896ae8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.961+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 shutdown_connections 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.961+0000 7f12d3c09740 1 -- 172.21.15.78:0/2031096428 wait complete. 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.961+0000 7f12d3c09740 20 client.15084 trim_cache size 0 max 16384 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 Processor -- start 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 -- start start 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae69d0 0x555896b01bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896b02960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555896b01a40 con 0x555896af1bb0 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555896b02110 con 0x555896ae72f0 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12d3c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555896b027e0 con 0x555896ae69d0 2024-09-16T01:10:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.978+0000 7f12cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34928/0 (socket says 172.21.15.78:34928) 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 -- 172.21.15.78:0/1189794581 learned_addr learned my addr 172.21.15.78:0/1189794581 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12caffd640 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae69d0 0x555896b01bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12c3fff640 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896b02960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 -- 172.21.15.78:0/1189794581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae69d0 msgr2=0x555896b01bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae69d0 0x555896b01bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 -- 172.21.15.78:0/1189794581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 msgr2=0x555896b02960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896b02960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 -- 172.21.15.78:0/1189794581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555896ae6800 con 0x555896af1bb0 2024-09-16T01:10:17.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12cb7fe640 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f12c4031580 tx=0x7f12c403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9c84d7133a115d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12c17fa640 1 -- 172.21.15.78:0/1189794581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4051030 con 0x555896af1bb0 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12c17fa640 1 -- 172.21.15.78:0/1189794581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c4045070 con 0x555896af1bb0 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.979+0000 7f12c17fa640 1 -- 172.21.15.78:0/1189794581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c404d480 con 0x555896af1bb0 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- 172.21.15.78:0/1189794581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 msgr2=0x555896af0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f12c4031580 tx=0x7f12c403c000 comp rx=0 tx=0).stop 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- 172.21.15.78:0/1189794581 shutdown_connections 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896af1bb0 0x555896af0790 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896ae72f0 0x555896b02960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 --2- 172.21.15.78:0/1189794581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae69d0 0x555896b01bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- 172.21.15.78:0/1189794581 >> 172.21.15.78:0/1189794581 conn(0x555896ae2f60 msgr2=0x555896ae3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:17.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- 172.21.15.78:0/1189794581 shutdown_connections 2024-09-16T01:10:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- 172.21.15.78:0/1189794581 wait complete. 2024-09-16T01:10:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 -- start start 2024-09-16T01:10:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.980+0000 7f12d3c09740 1 Processor -- start 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae72f0 0x555896b0c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896af1bb0 0x555896b07290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555896b03580 con 0x555896ae69d0 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555896b02eb0 con 0x555896af1bb0 2024-09-16T01:10:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555896b03c50 con 0x555896ae72f0 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12d3c09740 10 client.0 _ll_register_callbacks cb 0x7f12bc017630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34930/0 (socket says 172.21.15.78:34930) 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 -- 172.21.15.78:0/2028046944 learned_addr learned my addr 172.21.15.78:0/2028046944 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae72f0 0x555896b0c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 -- 172.21.15.78:0/2028046944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae72f0 msgr2=0x555896b0c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae72f0 0x555896b0c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 -- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896af1bb0 msgr2=0x555896b07290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896af1bb0 0x555896b07290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 -- 172.21.15.78:0/2028046944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555896b07ac0 con 0x555896ae69d0 2024-09-16T01:10:18.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12caffd640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f12bc09b310 tx=0x7f12bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fa71a8f08f5d74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.981+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc01e020 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f12bc09f400 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12c4045070 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f12bc0b3050 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 10 client.15090 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555896b6c600 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c4045070 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f12bc0979f0 con 0x555896ae69d0 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 10 client.15090 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 10 client.15090 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:10:18.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 20 client.15090 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b71030 0x555896b6fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 4 client.15090 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "status"}) -- 0x555896a54b10 con 0x555896b71030 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.982+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b71030 0x555896b6fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12c3fff640 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b71030 0x555896b6fd40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec1ef08116074ca0 server_cookie=49288d0ae5e6b7d2 in_seq=0 out_seq=0 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12aaffd640 10 client.15090 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12aaffd640 1 -- 172.21.15.78:0/2028046944 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555896a54b10 con 0x555896b71030 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12aaffd640 10 client.15090 handle_command_reply: tid=0 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 client.15090 shutdown 2024-09-16T01:10:18.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 msgr2=0x555896b0bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f12bc09b310 tx=0x7f12bc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 shutdown_connections 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555896b71030 0x555896b6fd40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555896af1bb0 0x555896b07290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555896ae72f0 0x555896b0c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12aaffd640 0 client.15090 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 --2- 172.21.15.78:0/2028046944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555896ae69d0 0x555896b0bd60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 >> 172.21.15.78:0/2028046944 conn(0x555896ae2f60 msgr2=0x555896ae2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 shutdown_connections 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 1 -- 172.21.15.78:0/2028046944 wait complete. 2024-09-16T01:10:18.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.983+0000 7f12d3c09740 20 client.15090 trim_cache size 0 max 16384 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 108.252277664, 2024-09-16T01:10:18.005 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:18.006 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-09-16T01:10:18.006 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:18.006 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 120.210752089 2024-09-16T01:10:18.006 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 msgr2=0x7f12b80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 0x7f12b80a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c40095c0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 msgr2=0x7f12cc0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00b4a0 tx=0x7f12bc00b960 comp rx=0 tx=0).stop 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 shutdown_connections 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f12b80a32d0 0x7f12b80a5790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12cc106f10 0x7f12cc074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12cc102790 0x7f12cc074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 --2- 172.21.15.78:0/2009522455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12cc101de0 0x7f12cc0744f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:18.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 >> 172.21.15.78:0/2009522455 conn(0x7f12cc0fd8c0 msgr2=0x7f12cc0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:18.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 shutdown_connections 2024-09-16T01:10:18.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:17.984+0000 7f12d1830640 1 -- 172.21.15.78:0/2009522455 wait complete. 2024-09-16T01:10:18.906 INFO:tasks.cephadm:Setup complete, yielding 2024-09-16T01:10:18.906 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-16T01:10:18.916 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:18.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-16T01:10:19.162 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:19.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:19 smithi078 ceph-mon[30937]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:19.348 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:19 smithi193 ceph-mon[35701]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:19 smithi148 ceph-mon[35030]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:19.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/3294458553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 msgr2=0x7f7e18102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:19.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/3294458553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18102000 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009900 tx=0x7f7e0802a410 comp rx=0 tx=0).stop 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/3294458553 shutdown_connections 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/3294458553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e181037c0 0x7f7e1810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/3294458553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e18102e00 0x7f7e18103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/3294458553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18102000 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/3294458553 >> 172.21.15.78:0/3294458553 conn(0x7f7e180fd3b0 msgr2=0x7f7e180ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/3294458553 shutdown_connections 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/3294458553 wait complete. 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.662+0000 7f7e1d9f6640 1 Processor -- start 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 -- start start 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e18102e00 0x7f7e18198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:19.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e181037c0 0x7f7e1819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810d860 con 0x7f7e18101c00 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810d6e0 con 0x7f7e18102e00 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1d9f6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1810d9e0 con 0x7f7e181037c0 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34946/0 (socket says 172.21.15.78:34946) 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 -- 172.21.15.78:0/1807525674 learned_addr learned my addr 172.21.15.78:0/1807525674 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e167fc640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e18102e00 0x7f7e18198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 -- 172.21.15.78:0/1807525674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e181037c0 msgr2=0x7f7e1819f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e181037c0 0x7f7e1819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 -- 172.21.15.78:0/1807525674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e18102e00 msgr2=0x7f7e18198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e18102e00 0x7f7e18198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 -- 172.21.15.78:0/1807525674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1819fb10 con 0x7f7e18101c00 2024-09-16T01:10:19.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e16ffd640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e08031610 tx=0x7f7e080043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.663+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0803f030 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.664+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e08003ce0 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.664+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1819fda0 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.664+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08035070 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.664+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e181a01c0 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.665+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e18109050 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.666+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7e08042030 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.666+0000 7f7e1c9f4640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 0x7f7e000a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.666+0000 7f7e167fc640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 0x7f7e000a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.666+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7e080f0430 con 0x7f7e18101c00 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.667+0000 7f7e167fc640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 0x7f7e000a57c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:19.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.670+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7e080bcc90 con 0x7f7e18101c00 2024-09-16T01:10:19.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.855+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f7e18062630 con 0x7f7e000a3300 2024-09-16T01:10:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.856+0000 7f7e1c9f4640 1 -- 172.21.15.78:0/1807525674 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f7e18062630 con 0x7f7e000a3300 2024-09-16T01:10:19.857 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2024-09-16T01:10:19.857 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2024-09-16T01:10:19.857 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 msgr2=0x7f7e000a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 0x7f7e000a57c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c005850 comp rx=0 tx=0).stop 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 msgr2=0x7f7e18197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e08031610 tx=0x7f7e080043b0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 shutdown_connections 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f7e000a3300 0x7f7e000a57c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e181037c0 0x7f7e1819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e18102e00 0x7f7e18198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 --2- 172.21.15.78:0/1807525674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e18101c00 0x7f7e18197ee0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:19.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 >> 172.21.15.78:0/1807525674 conn(0x7f7e180fd3b0 msgr2=0x7f7e18106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:19.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 shutdown_connections 2024-09-16T01:10:19.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:19.858+0000 7f7e1d9f6640 1 -- 172.21.15.78:0/1807525674 wait complete. 2024-09-16T01:10:20.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-16T01:10:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:21 smithi078 ceph-mon[30937]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:21.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:21 smithi078 ceph-mon[30937]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:21 smithi148 ceph-mon[35030]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:21 smithi148 ceph-mon[35030]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 -- 172.21.15.78:0/2569094282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64181078c0 msgr2=0x7f6418109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 --2- 172.21.15.78:0/2569094282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64181078c0 0x7f6418109d10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6410009ca0 tx=0x7f641002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 -- 172.21.15.78:0/2569094282 shutdown_connections 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 --2- 172.21.15.78:0/2569094282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64181078c0 0x7f6418109d10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 --2- 172.21.15.78:0/2569094282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 0x7f6418107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 --2- 172.21.15.78:0/2569094282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6418102660 0x7f6418104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 -- 172.21.15.78:0/2569094282 >> 172.21.15.78:0/2569094282 conn(0x7f64180fc280 msgr2=0x7f64180fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 -- 172.21.15.78:0/2569094282 shutdown_connections 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.413+0000 7f641f936640 1 -- 172.21.15.78:0/2569094282 wait complete. 2024-09-16T01:10:21.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 Processor -- start 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 -- start start 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 0x7f6418198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64181078c0 0x7f641819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f641810d880 con 0x7f6418102660 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f641810d700 con 0x7f6418104f90 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641f936640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f641810da00 con 0x7f64181078c0 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641d6d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641d6d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34962/0 (socket says 172.21.15.78:34962) 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641d6d2640 1 -- 172.21.15.78:0/2504815534 learned_addr learned my addr 172.21.15.78:0/2504815534 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641ced1640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 0x7f6418198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:21.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.414+0000 7f641d6d2640 1 -- 172.21.15.78:0/2504815534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64181078c0 msgr2=0x7f641819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641d6d2640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64181078c0 0x7f641819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641d6d2640 1 -- 172.21.15.78:0/2504815534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 msgr2=0x7f6418198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641d6d2640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 0x7f6418198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641d6d2640 1 -- 172.21.15.78:0/2504815534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641819fb00 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641d6d2640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f641400d770 tx=0x7f641400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414017070 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641819fdf0 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6414005500 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414004e00 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.415+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64181a0250 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.416+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64180fdd70 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.417+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6414004fa0 con 0x7f6418102660 2024-09-16T01:10:21.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.418+0000 7f640e7fc640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 0x7f64040a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.418+0000 7f641ced1640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 0x7f64040a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.418+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f64140cfcc0 con 0x7f6418102660 2024-09-16T01:10:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.418+0000 7f641ced1640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 0x7f64040a57c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6408005fd0 tx=0x7f6408005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:21.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.421+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f641409c520 con 0x7f6418102660 2024-09-16T01:10:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:21 smithi193 ceph-mon[35701]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:21 smithi193 ceph-mon[35701]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:21.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.595+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6418062630 con 0x7f64040a3300 2024-09-16T01:10:21.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.607+0000 7f640e7fc640 1 -- 172.21.15.78:0/2504815534 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4061 (secure 0 0 0) 0x7f6418062630 con 0x7f64040a3300 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (2m) 55s ago 2m 14.0M - 19.1.1-326-g8921c8ef f1455bd0219c 1b8d9272c81f 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi148 running (2m) 57s ago 2m 26.2M - 19.1.1-326-g8921c8ef f1455bd0219c 8f3a27f8eb92 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi193 running (2m) 57s ago 2m 16.7M - 19.1.1-326-g8921c8ef f1455bd0219c cb7b57758f1e 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (2m) 55s ago 2m 13.4M - 19.1.1-326-g8921c8ef f1455bd0219c 8a40bcc7f2b4 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi148 running (2m) 57s ago 2m 14.7M - 19.1.1-326-g8921c8ef f1455bd0219c 2d1c85e410e0 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi193 running (2m) 57s ago 2m 17.9M - 19.1.1-326-g8921c8ef f1455bd0219c 3c2b91514f24 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (2m) 55s ago 2m 16.6M - 19.1.1-326-g8921c8ef f1455bd0219c 14980425d884 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi148 running (2m) 57s ago 2m 18.9M - 19.1.1-326-g8921c8ef f1455bd0219c f60a393650bb 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi193 running (2m) 57s ago 2m 17.5M - 19.1.1-326-g8921c8ef f1455bd0219c 1699aba69888 2024-09-16T01:10:21.609 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (2m) 55s ago 2m 16.7M - 19.1.1-326-g8921c8ef f1455bd0219c 523cf01e8a22 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi148 running (2m) 57s ago 2m 14.4M - 19.1.1-326-g8921c8ef f1455bd0219c 891e4f2530e1 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi193 running (2m) 57s ago 2m 14.2M - 19.1.1-326-g8921c8ef f1455bd0219c 5ac6618a7df0 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (9m) 55s ago 9m 509M - 19.1.1-326-g8921c8ef f1455bd0219c 12677b9a0f03 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi148 *:8443,8765 running (7m) 57s ago 7m 442M - 19.1.1-326-g8921c8ef f1455bd0219c 639eed3ea344 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi193 *:8443,8765 running (7m) 57s ago 7m 442M - 19.1.1-326-g8921c8ef f1455bd0219c 1822b141934b 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (9m) 55s ago 9m 51.0M 2048M 19.1.1-326-g8921c8ef f1455bd0219c edc50c2d55b7 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi148 running (8m) 57s ago 8m 42.9M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 517650b49843 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi193 running (8m) 57s ago 8m 39.7M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 18b55d000551 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (7m) 55s ago 7m 55.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c fffbae5ba777 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi148 running (7m) 57s ago 7m 58.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c dea72cdb0df7 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi193 running (6m) 57s ago 6m 47.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 9b5c4e3651a8 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (6m) 55s ago 6m 54.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 16f9c7cfcf21 2024-09-16T01:10:21.610 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi148 running (5m) 57s ago 5m 58.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c ec6fd1dafcb0 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi193 running (5m) 57s ago 5m 55.7M 4272M 19.1.1-326-g8921c8ef f1455bd0219c ed3ca084fa2d 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (4m) 55s ago 4m 43.5M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 557d18ee422f 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi148 running (4m) 57s ago 4m 56.1M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 604992f5b581 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi193 running (4m) 57s ago 4m 43.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 049f75ad26b4 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (3m) 55s ago 3m 53.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c d0841ad4c88d 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi148 running (3m) 57s ago 3m 57.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c e65ec22035c5 2024-09-16T01:10:21.611 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi193 running (2m) 57s ago 2m 40.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c de614862893f 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 msgr2=0x7f64040a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 0x7f64040a57c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6408005fd0 tx=0x7f6408005850 comp rx=0 tx=0).stop 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 msgr2=0x7f6418197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f641400d770 tx=0x7f641400dc30 comp rx=0 tx=0).stop 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 shutdown_connections 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f64040a3300 0x7f64040a57c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64181078c0 0x7f641819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6418104f90 0x7f6418198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 --2- 172.21.15.78:0/2504815534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6418102660 0x7f6418197ed0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 >> 172.21.15.78:0/2504815534 conn(0x7f64180fc280 msgr2=0x7f6418102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 shutdown_connections 2024-09-16T01:10:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:21.610+0000 7f641f936640 1 -- 172.21.15.78:0/2504815534 wait complete. 2024-09-16T01:10:22.482 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-16T01:10:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:23.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:23 smithi078 ceph-mon[30937]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:23.246 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:23 smithi078 ceph-mon[30937]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 -- 172.21.15.78:0/301002554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254101e30 msgr2=0x7f82541086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:23.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 --2- 172.21.15.78:0/301002554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254101e30 0x7f82541086c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f824c009ca0 tx=0x7f824c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 -- 172.21.15.78:0/301002554 shutdown_connections 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 --2- 172.21.15.78:0/301002554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254101e30 0x7f82541086c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 --2- 172.21.15.78:0/301002554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 0x7f82541018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 --2- 172.21.15.78:0/301002554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8254100270 0x7f8254100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 -- 172.21.15.78:0/301002554 >> 172.21.15.78:0/301002554 conn(0x7f82540fba40 msgr2=0x7f82540fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 -- 172.21.15.78:0/301002554 shutdown_connections 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.323+0000 7f825b669640 1 -- 172.21.15.78:0/301002554 wait complete. 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 Processor -- start 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 -- start start 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 0x7f825419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8254101e30 0x7f825419f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:23.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f825410bf10 con 0x7f8254100270 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f825410bd90 con 0x7f8254101e30 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f825b669640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f825410c090 con 0x7f8254101470 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34974/0 (socket says 172.21.15.78:34974) 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 -- 172.21.15.78:0/348029739 learned_addr learned my addr 172.21.15.78:0/348029739 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8258c04640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 0x7f825419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 -- 172.21.15.78:0/348029739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 msgr2=0x7f825419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 0x7f825419cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 -- 172.21.15.78:0/348029739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8254101e30 msgr2=0x7f825419f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8254101e30 0x7f825419f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.324+0000 7f8259405640 1 -- 172.21.15.78:0/348029739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825419fb40 con 0x7f8254100270 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f8259405640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f825000c970 tx=0x7f825000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:23.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250004220 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82500043c0 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825000f450 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825419fdd0 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.325+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82541a01d0 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.326+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82541a0490 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.327+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f82500079b0 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.328+0000 7f824a7fc640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 0x7f82400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.328+0000 7f8258c04640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 0x7f82400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.328+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f82500cfc50 con 0x7f8254100270 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.328+0000 7f8258c04640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 0x7f82400a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f82540fe120 tx=0x7f8244005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.332+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8250014030 con 0x7f8254100270 2024-09-16T01:10:23.581 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:23 smithi193 ceph-mon[35701]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:23.581 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:23 smithi193 ceph-mon[35701]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:23.581 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:23 smithi148 ceph-mon[35030]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:23.582 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:23 smithi148 ceph-mon[35030]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:23.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.506+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8254062630 con 0x7f82400a32d0 2024-09-16T01:10:23.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.512+0000 7f824a7fc640 1 -- 172.21.15.78:0/348029739 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8254062630 con 0x7f82400a32d0 2024-09-16T01:10:23.582 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-16T01:10:23.582 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 59s ago 2m smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi148=b;smithi148=e;smithi148=h;smithi148=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-09-16T01:10:23.582 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 59s ago 7m smithi078=x;smithi148=y;smithi193=z;count:3 2024-09-16T01:10:23.583 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 59s ago 8m smithi078:172.21.15.78=a;smithi148:172.21.15.148=b;smithi193:172.21.15.193=c;count:3 2024-09-16T01:10:23.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 59s ago - 2024-09-16T01:10:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 msgr2=0x7f82400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 0x7f82400a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f82540fe120 tx=0x7f8244005c60 comp rx=0 tx=0).stop 2024-09-16T01:10:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 msgr2=0x7f82540fdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f825000c970 tx=0x7f825000ce30 comp rx=0 tx=0).stop 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 shutdown_connections 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f82400a32d0 0x7f82400a5790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8254101e30 0x7f825419f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8254101470 0x7f825419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 --2- 172.21.15.78:0/348029739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8254100270 0x7f82540fdba0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 >> 172.21.15.78:0/348029739 conn(0x7f82540fba40 msgr2=0x7f8254104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.514+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 shutdown_connections 2024-09-16T01:10:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:23.515+0000 7f825b669640 1 -- 172.21.15.78:0/348029739 wait complete. 2024-09-16T01:10:24.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-16T01:10:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 -- 172.21.15.78:0/3714576247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634102e00 msgr2=0x7f6634103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 --2- 172.21.15.78:0/3714576247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634102e00 0x7f6634103280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6630009900 tx=0x7f663002a410 comp rx=0 tx=0).stop 2024-09-16T01:10:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 -- 172.21.15.78:0/3714576247 shutdown_connections 2024-09-16T01:10:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 --2- 172.21.15.78:0/3714576247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66341037c0 0x7f663410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 --2- 172.21.15.78:0/3714576247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634102e00 0x7f6634103280 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 --2- 172.21.15.78:0/3714576247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6634101c00 0x7f6634102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 -- 172.21.15.78:0/3714576247 >> 172.21.15.78:0/3714576247 conn(0x7f66340fd390 msgr2=0x7f66340ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.068+0000 7f663b70d640 1 -- 172.21.15.78:0/3714576247 shutdown_connections 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 -- 172.21.15.78:0/3714576247 wait complete. 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 Processor -- start 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 -- start start 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 0x7f6634198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 0x7f663419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f663410d980 con 0x7f6634101c00 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f663410d800 con 0x7f6634102e00 2024-09-16T01:10:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f663b70d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f663410db00 con 0x7f66341037c0 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f66394a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f66394a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59562/0 (socket says 172.21.15.78:59562) 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.069+0000 7f66394a9640 1 -- 172.21.15.78:0/3003943927 learned_addr learned my addr 172.21.15.78:0/3003943927 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f6638ca8640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 0x7f6634198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f6639caa640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 0x7f663419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 -- 172.21.15.78:0/3003943927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 msgr2=0x7f663419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 0x7f663419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 -- 172.21.15.78:0/3003943927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 msgr2=0x7f6634198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 0x7f6634198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 -- 172.21.15.78:0/3003943927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663419fcf0 con 0x7f6634101c00 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f6639caa640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 0x7f663419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f6638ca8640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 0x7f6634198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f66394a9640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f662400b450 tx=0x7f662400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:25.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624015020 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6624011d70 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663419ffe0 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662400f340 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.070+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66341a0460 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.071+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6634109140 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.072+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6624018020 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.073+0000 7f662a7fc640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 0x7f66200a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.073+0000 7f6638ca8640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 0x7f66200a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.073+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f66240cf850 con 0x7f6634101c00 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.073+0000 7f6638ca8640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 0x7f66200a57c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6630000c00 tx=0x7f66300315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:25.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.077+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f662401f020 con 0x7f6634101c00 2024-09-16T01:10:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.251+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6634062630 con 0x7f66200a3300 2024-09-16T01:10:25.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:25 smithi078 ceph-mon[30937]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:25.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:25 smithi078 ceph-mon[30937]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:25.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.253+0000 7f662a7fc640 1 -- 172.21.15.78:0/3003943927 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6634062630 con 0x7f66200a3300 2024-09-16T01:10:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2024-09-16T01:10:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2024-09-16T01:10:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:smithi148 172.21.15.148 2024-09-16T01:10:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:smithi193 172.21.15.193 2024-09-16T01:10:25.254 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 msgr2=0x7f66200a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 0x7f66200a57c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6630000c00 tx=0x7f66300315d0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 msgr2=0x7f66341980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f662400b450 tx=0x7f662400b910 comp rx=0 tx=0).stop 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 shutdown_connections 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f66200a3300 0x7f66200a57c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66341037c0 0x7f663419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6634102e00 0x7f6634198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 --2- 172.21.15.78:0/3003943927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6634101c00 0x7f66341980c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:25.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 >> 172.21.15.78:0/3003943927 conn(0x7f66340fd390 msgr2=0x7f6634106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:25.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 shutdown_connections 2024-09-16T01:10:25.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:25.255+0000 7f663b70d640 1 -- 172.21.15.78:0/3003943927 wait complete. 2024-09-16T01:10:25.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:25 smithi148 ceph-mon[35030]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:25 smithi148 ceph-mon[35030]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:25 smithi193 ceph-mon[35701]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:25 smithi193 ceph-mon[35701]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:25.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-16T01:10:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:26.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- 172.21.15.78:0/1937724930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554102690 msgr2=0x7fd554108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:26.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 --2- 172.21.15.78:0/1937724930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554102690 0x7fd554108f20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009ca0 tx=0x7fd54c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- 172.21.15.78:0/1937724930 shutdown_connections 2024-09-16T01:10:26.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 --2- 172.21.15.78:0/1937724930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554102690 0x7fd554108f20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 --2- 172.21.15.78:0/1937724930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 --2- 172.21.15.78:0/1937724930 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554100ad0 0x7fd554100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- 172.21.15.78:0/1937724930 >> 172.21.15.78:0/1937724930 conn(0x7fd5540fc280 msgr2=0x7fd5540fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- 172.21.15.78:0/1937724930 shutdown_connections 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- 172.21.15.78:0/1937724930 wait complete. 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 Processor -- start 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.746+0000 7fd55a938640 1 -- start start 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 0x7fd55410b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554102690 0x7fd554109fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd55410d880 con 0x7fd554100ad0 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd55410d700 con 0x7fd554102690 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd55a938640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd55410da00 con 0x7fd554101cd0 2024-09-16T01:10:26.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34700/0 (socket says 172.21.15.78:34700) 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd5537fe640 1 -- 172.21.15.78:0/3846534584 learned_addr learned my addr 172.21.15.78:0/3846534584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 0x7fd55410b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd558ed5640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554102690 0x7fd554109fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 -- 172.21.15.78:0/3846534584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 msgr2=0x7fd554109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554109aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 -- 172.21.15.78:0/3846534584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554102690 msgr2=0x7fd554109fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554102690 0x7fd554109fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 -- 172.21.15.78:0/3846534584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55410a520 con 0x7fd554100ad0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.747+0000 7fd553fff640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 0x7fd55410b450 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c940 tx=0x7fd54400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.748+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544007bb0 con 0x7fd554100ad0 2024-09-16T01:10:26.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.748+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd544007d50 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.748+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55410a810 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.748+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544018790 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.748+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5541a6900 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.749+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554100ed0 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.750+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd54400f450 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.750+0000 7fd5517fa640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 0x7fd53c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.750+0000 7fd5537fe640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 0x7fd53c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.750+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5440cfdb0 con 0x7fd554100ad0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.751+0000 7fd5537fe640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 0x7fd53c0a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd55410b1d0 tx=0x7fd54800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:26.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.755+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd544014030 con 0x7fd554100ad0 2024-09-16T01:10:26.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.951+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fd554062630 con 0x7fd53c0a32d0 2024-09-16T01:10:26.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.952+0000 7fd5517fa640 1 -- 172.21.15.78:0/3846534584 <== mgr.14150 v2:172.21.15.78:6800/1698663383 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fd554062630 con 0x7fd53c0a32d0 2024-09-16T01:10:26.953 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-16T01:10:26.953 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-16T01:10:26.953 INFO:teuthology.orchestra.run.smithi078.stdout:smithi148 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233005G400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-16T01:10:26.953 INFO:teuthology.orchestra.run.smithi078.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.954+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 msgr2=0x7fd53c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.954+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 0x7fd53c0a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd55410b1d0 tx=0x7fd54800b410 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.954+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 msgr2=0x7fd55410b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.954+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 0x7fd55410b450 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c940 tx=0x7fd54400ce00 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 shutdown_connections 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd53c0a32d0 0x7fd53c0a5790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd554102690 0x7fd554109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd554101cd0 0x7fd554109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 --2- 172.21.15.78:0/3846534584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd554100ad0 0x7fd55410b450 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 >> 172.21.15.78:0/3846534584 conn(0x7fd5540fc280 msgr2=0x7fd554104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 shutdown_connections 2024-09-16T01:10:26.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:26.955+0000 7fd55a938640 1 -- 172.21.15.78:0/3846534584 wait complete. 2024-09-16T01:10:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:27 smithi078 ceph-mon[30937]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:27 smithi078 ceph-mon[30937]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:27 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:27 smithi148 ceph-mon[35030]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:27 smithi148 ceph-mon[35030]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:27 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:27 smithi193 ceph-mon[35701]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:27 smithi193 ceph-mon[35701]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:27 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:28.104 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-16T01:10:28.113 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:28.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-16T01:10:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:28 smithi148 ceph-mon[35030]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:28 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:28 smithi078 ceph-mon[30937]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:28 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:28 smithi193 ceph-mon[35701]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:28.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:28 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:28.598 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- 172.21.15.78:0/1921208206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c1026d0 msgr2=0x7f034c108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 --2- 172.21.15.78:0/1921208206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c1026d0 0x7f034c108f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0344009ca0 tx=0x7f034402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- 172.21.15.78:0/1921208206 shutdown_connections 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 --2- 172.21.15.78:0/1921208206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c1026d0 0x7f034c108f60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 --2- 172.21.15.78:0/1921208206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101d10 0x7f034c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 --2- 172.21.15.78:0/1921208206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c100b10 0x7f034c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- 172.21.15.78:0/1921208206 >> 172.21.15.78:0/1921208206 conn(0x7f034c0fc2a0 msgr2=0x7f034c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- 172.21.15.78:0/1921208206 shutdown_connections 2024-09-16T01:10:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- 172.21.15.78:0/1921208206 wait complete. 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 Processor -- start 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.068+0000 7f0352a36640 1 -- start start 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 0x7f034c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c1026d0 0x7f034c19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10c860 con 0x7f034c100b10 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10c6e0 con 0x7f034c101d10 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0352a36640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10c9e0 con 0x7f034c1026d0 2024-09-16T01:10:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59600/0 (socket says 172.21.15.78:59600) 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 -- 172.21.15.78:0/2071763030 learned_addr learned my addr 172.21.15.78:0/2071763030 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f0350fd3640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c1026d0 0x7f034c19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034b7fe640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 0x7f034c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 -- 172.21.15.78:0/2071763030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c1026d0 msgr2=0x7f034c19f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c1026d0 0x7f034c19f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 -- 172.21.15.78:0/2071763030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 msgr2=0x7f034c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 0x7f034c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 -- 172.21.15.78:0/2071763030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c19fb20 con 0x7f034c100b10 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034b7fe640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 0x7f034c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:29.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f034bfff640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f033c00b450 tx=0x7f033c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.069+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c015020 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.070+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c011d70 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.070+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c19fdb0 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.070+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c00f340 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.070+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c1a02e0 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.071+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c107e40 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.072+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f033c018020 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.072+0000 7f03497fa640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 0x7f03340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.073+0000 7f034b7fe640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 0x7f03340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.073+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f033c0cf770 con 0x7f034c100b10 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.073+0000 7f034b7fe640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 0x7f03340a5790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f034c06b870 tx=0x7f0340005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.077+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f033c01f020 con 0x7f034c100b10 2024-09-16T01:10:29.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:29 smithi078 ceph-mon[30937]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.305+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f034c02da10 con 0x7f034c100b10 2024-09-16T01:10:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.305+0000 7f03497fa640 1 -- 172.21.15.78:0/2071763030 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3092 (secure 0 0 0) 0x7f033c00d040 con 0x7f034c100b10 2024-09-16T01:10:29.306 INFO:teuthology.orchestra.run.smithi078.stdout:e24 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:btime 2024-09-16T01:09:08:102275+0000 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 24 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-09-16T01:08:29.385380+0000 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-09-16T01:09:07.935774+0000 2024-09-16T01:10:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2024-09-16T01:10:29.308 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24485,1=24460,2=24455} 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3,4] 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:qdb_cluster leader: 24485 members: 24455,24460,24485 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24485} state up:active seq 4 addr [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.193:6836/1637418104,v1:172.21.15.193:6837/1637418104] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{1:24460} state up:active seq 15 addr [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{1:24436} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/1850679548,v1:172.21.15.193:6835/1850679548] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{2:24455} state up:active seq 16 addr [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{2:24434} state up:standby-replay seq 1 addr [v2:172.21.15.148:6834/1399158659,v1:172.21.15.148:6835/1399158659] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:14526} state up:standby seq 1 addr [v2:172.21.15.78:6834/557283089,v1:172.21.15.78:6835/557283089] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14538} state up:standby seq 1 addr [v2:172.21.15.78:6836/4143414338,v1:172.21.15.78:6837/4143414338] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{-1:14580} state up:standby seq 1 addr [v2:172.21.15.78:6838/2440325522,v1:172.21.15.78:6839/2440325522] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14592} state up:standby seq 1 addr [v2:172.21.15.78:6840/643835923,v1:172.21.15.78:6841/643835923] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24397} state up:standby seq 1 addr [v2:172.21.15.193:6832/4092784262,v1:172.21.15.193:6833/4092784262] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.310 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24413} state up:standby seq 1 addr [v2:172.21.15.148:6832/909591038,v1:172.21.15.148:6833/909591038] compat {c=[1],r=[1],i=[1fff]}] 2024-09-16T01:10:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.308+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 msgr2=0x7f03340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.308+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 0x7f03340a5790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f034c06b870 tx=0x7f0340005d90 comp rx=0 tx=0).stop 2024-09-16T01:10:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.308+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 msgr2=0x7f034c06b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.308+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f033c00b450 tx=0x7f033c00b910 comp rx=0 tx=0).stop 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 shutdown_connections 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f03340a32d0 0x7f03340a5790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c1026d0 0x7f034c19f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f034c101d10 0x7f034c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 --2- 172.21.15.78:0/2071763030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f034c100b10 0x7f034c06b2f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 >> 172.21.15.78:0/2071763030 conn(0x7f034c0fc2a0 msgr2=0x7f034c105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 shutdown_connections 2024-09-16T01:10:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:29.309+0000 7f0352a36640 1 -- 172.21.15.78:0/2071763030 wait complete. 2024-09-16T01:10:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:29 smithi148 ceph-mon[35030]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:29 smithi193 ceph-mon[35701]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:30.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-16T01:10:30.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2071763030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T01:10:30.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:30.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:30.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:30 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:30.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2071763030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T01:10:30.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.512 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:30.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:30.513 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:30 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2071763030' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:30.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:30 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' 2024-09-16T01:10:30.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 -- 172.21.15.78:0/1454737484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c100b10 msgr2=0x7fdb3c100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:30.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/1454737484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c100b10 0x7fdb3c100f10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdb30009900 tx=0x7fdb3002a410 comp rx=0 tx=0).stop 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 -- 172.21.15.78:0/1454737484 shutdown_connections 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/1454737484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 0x7fdb3c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/1454737484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c101d10 0x7fdb3c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/1454737484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c100b10 0x7fdb3c100f10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 -- 172.21.15.78:0/1454737484 >> 172.21.15.78:0/1454737484 conn(0x7fdb3c0fc2a0 msgr2=0x7fdb3c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 -- 172.21.15.78:0/1454737484 shutdown_connections 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 -- 172.21.15.78:0/1454737484 wait complete. 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.906+0000 7fdb42ed6640 1 Processor -- start 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 -- start start 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c100b10 0x7fdb3c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:30.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 0x7fdb3c19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c10c7f0 con 0x7fdb3c101d10 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c10c670 con 0x7fdb3c1026d0 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb42ed6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c10c970 con 0x7fdb3c100b10 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59610/0 (socket says 172.21.15.78:59610) 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 -- 172.21.15.78:0/3712005986 learned_addr learned my addr 172.21.15.78:0/3712005986 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb40c72640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c100b10 0x7fdb3c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 -- 172.21.15.78:0/3712005986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c100b10 msgr2=0x7fdb3c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb41473640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 0x7fdb3c19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c100b10 0x7fdb3c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 -- 172.21.15.78:0/3712005986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 msgr2=0x7fdb3c19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:30.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 0x7fdb3c19f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 -- 172.21.15.78:0/3712005986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c19fb60 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb3bfff640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c920 tx=0x7fdb2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.907+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28007c70 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.908+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c19fe50 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.908+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb28007e10 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.908+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28018890 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.908+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3c1a0380 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.909+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb3c1a0610 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.910+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdb28018a30 con 0x7fdb3c101d10 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.910+0000 7fdb39ffb640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 0x7fdb2c0a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.910+0000 7fdb40c72640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 0x7fdb2c0a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:30.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.910+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdb280cffd0 con 0x7fdb3c101d10 2024-09-16T01:10:30.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.911+0000 7fdb40c72640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 0x7fdb2c0a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb30031610 tx=0x7fdb30000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:30.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:30.914+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdb28014030 con 0x7fdb3c101d10 2024-09-16T01:10:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.084+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fdb3c1a07e0 con 0x7fdb3c101d10 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.085+0000 7fdb39ffb640 1 -- 172.21.15.78:0/3712005986 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5584 (secure 0 0 0) 0x7fdb2800de20 con 0x7fdb3c101d10 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 91 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:fsid fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-09-16T01:00:45.346692+0000 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-09-16T01:09:39.105086+0000 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T01:10:31.086 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 27 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release squid 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-16T01:10:31.087 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/3172670933,v1:172.21.15.78:6803/3172670933] [v2:172.21.15.78:6804/3172670933,v1:172.21.15.78:6805/3172670933] exists,up ded124f4-818a-4255-a9cb-dcaa0fc71556 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6800/593162683,v1:172.21.15.148:6801/593162683] [v2:172.21.15.148:6802/593162683,v1:172.21.15.148:6803/593162683] exists,up 82da4924-eaf5-44df-adce-7f605462c130 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/3453663394,v1:172.21.15.193:6801/3453663394] [v2:172.21.15.193:6802/3453663394,v1:172.21.15.193:6803/3453663394] exists,up 51654cac-6b84-4b81-9861-6218a4f82eab 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/419734598,v1:172.21.15.78:6811/419734598] [v2:172.21.15.78:6812/419734598,v1:172.21.15.78:6813/419734598] exists,up 3aa4ba88-9fb8-4dfc-bc12-f313c953351a 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 32 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6808/2237990015,v1:172.21.15.148:6809/2237990015] [v2:172.21.15.148:6810/2237990015,v1:172.21.15.148:6811/2237990015] exists,up b8411f2d-a262-4113-95f9-ca6ae72ab967 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/2281160557,v1:172.21.15.193:6809/2281160557] [v2:172.21.15.193:6810/2281160557,v1:172.21.15.193:6811/2281160557] exists,up 3508afd5-2958-42dd-81c0-5f50be889ec6 2024-09-16T01:10:31.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 45 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/3932044692,v1:172.21.15.78:6819/3932044692] [v2:172.21.15.78:6820/3932044692,v1:172.21.15.78:6821/3932044692] exists,up 0c2c983a-999e-45fe-bd9e-270546696ff2 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6816/883929363,v1:172.21.15.148:6817/883929363] [v2:172.21.15.148:6818/883929363,v1:172.21.15.148:6819/883929363] exists,up c9855f37-2fd8-44a6-a6d9-80cf978e357a 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 58 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/692862816,v1:172.21.15.193:6817/692862816] [v2:172.21.15.193:6818/692862816,v1:172.21.15.193:6819/692862816] exists,up be96be1e-a654-46f9-a1aa-d2ce999d7a67 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 65 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/409695790,v1:172.21.15.78:6827/409695790] [v2:172.21.15.78:6828/409695790,v1:172.21.15.78:6829/409695790] exists,up fc303551-6913-47a4-ade8-be306c49a4a3 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 70 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6824/85363667,v1:172.21.15.148:6825/85363667] [v2:172.21.15.148:6826/85363667,v1:172.21.15.148:6827/85363667] exists,up ae7b8367-0244-4150-97f6-8ff99149e440 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 77 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/1055668594,v1:172.21.15.193:6825/1055668594] [v2:172.21.15.193:6826/1055668594,v1:172.21.15.193:6827/1055668594] exists,up e9f87f81-392a-4490-9a01-394b494b5251 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3 [4,9] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.e [5,9] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.11 [4,8] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.14 [4,6] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.20 [5,9] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.28 [3,8] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.2 [9,1] 2024-09-16T01:10:31.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.6 [9,11] 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.12 [5,10] 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.13 [9,8] 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/238352864 expires 2024-09-17T01:01:37.912173+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/4128193457 expires 2024-09-17T01:01:37.912173+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/2425087015 expires 2024-09-17T01:01:37.912173+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/3073396374 expires 2024-09-17T01:01:12.814641+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2021750195 expires 2024-09-17T01:01:37.912173+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/3073396374 expires 2024-09-17T01:01:12.814641+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/3690753154 expires 2024-09-17T01:01:12.814641+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/2425087015 expires 2024-09-17T01:01:37.912173+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1014137394 expires 2024-09-17T01:01:12.814641+0000 2024-09-16T01:10:31.090 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/65187856 expires 2024-09-17T01:01:12.814641+0000 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 msgr2=0x7fdb2c0a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 0x7fdb2c0a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb30031610 tx=0x7fdb30000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 msgr2=0x7fdb3c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c920 tx=0x7fdb2800cde0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 shutdown_connections 2024-09-16T01:10:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fdb2c0a3300 0x7fdb2c0a57c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1026d0 0x7fdb3c19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb3c101d10 0x7fdb3c198470 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 --2- 172.21.15.78:0/3712005986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb3c100b10 0x7fdb3c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 >> 172.21.15.78:0/3712005986 conn(0x7fdb3c0fc2a0 msgr2=0x7fdb3c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 shutdown_connections 2024-09-16T01:10:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:31.087+0000 7fdb42ed6640 1 -- 172.21.15.78:0/3712005986 wait complete. 2024-09-16T01:10:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:31 smithi148 ceph-mon[35030]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:31.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:31 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3712005986' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T01:10:31.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:31 smithi078 ceph-mon[30937]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:31.455 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:31 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3712005986' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T01:10:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:31 smithi193 ceph-mon[35701]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-16T01:10:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:31 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3712005986' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-16T01:10:31.813 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-16T01:10:31.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:32.075 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 -- 172.21.15.78:0/1414173038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 msgr2=0x7fec90100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 --2- 172.21.15.78:0/1414173038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90100ee0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec80009930 tx=0x7fec8002a420 comp rx=0 tx=0).stop 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 -- 172.21.15.78:0/1414173038 shutdown_connections 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 --2- 172.21.15.78:0/1414173038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90101420 0x7fec9010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 --2- 172.21.15.78:0/1414173038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90100ee0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 --2- 172.21.15.78:0/1414173038 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90106a80 0x7fec90106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 -- 172.21.15.78:0/1414173038 >> 172.21.15.78:0/1414173038 conn(0x7fec900fc7a0 msgr2=0x7fec900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 -- 172.21.15.78:0/1414173038 shutdown_connections 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.645+0000 7fec97857640 1 -- 172.21.15.78:0/1414173038 wait complete. 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 Processor -- start 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 -- start start 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90101420 0x7fec901989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90106a80 0x7fec9019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fec90110ea0 con 0x7fec90100a80 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fec90110d20 con 0x7fec90101420 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec97857640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec90111020 con 0x7fec90106a80 2024-09-16T01:10:32.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec955f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec955f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59622/0 (socket says 172.21.15.78:59622) 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec955f3640 1 -- 172.21.15.78:0/418007232 learned_addr learned my addr 172.21.15.78:0/418007232 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.646+0000 7fec94df2640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90101420 0x7fec901989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec95df4640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90106a80 0x7fec9019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 -- 172.21.15.78:0/418007232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90106a80 msgr2=0x7fec9019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90106a80 0x7fec9019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 -- 172.21.15.78:0/418007232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90101420 msgr2=0x7fec901989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90101420 0x7fec901989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 -- 172.21.15.78:0/418007232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9019d420 con 0x7fec90100a80 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec955f3640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00b4a0 tx=0x7fec8c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c014070 con 0x7fec90100a80 2024-09-16T01:10:32.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9019d710 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec8c010af0 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c017d10 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.647+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec901a4ff0 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.648+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90100ee0 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.649+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fec8c017070 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.650+0000 7fec867fc640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 0x7fec7c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.650+0000 7fec94df2640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 0x7fec7c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.650+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec8c0cf170 con 0x7fec90100a80 2024-09-16T01:10:32.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.650+0000 7fec94df2640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 0x7fec7c0a5790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:32.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.654+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fec8c09bae0 con 0x7fec90100a80 2024-09-16T01:10:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.854+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec9002d7c0 con 0x7fec90100a80 2024-09-16T01:10:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.855+0000 7fec867fc640 1 -- 172.21.15.78:0/418007232 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7fec8c09b3e0 con 0x7fec90100a80 2024-09-16T01:10:32.856 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:32.856 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:10:32.856 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 msgr2=0x7fec7c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 0x7fec7c0a5790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80031490 comp rx=0 tx=0).stop 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 msgr2=0x7fec90198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00b4a0 tx=0x7fec8c00b960 comp rx=0 tx=0).stop 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 shutdown_connections 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fec7c0a32d0 0x7fec7c0a5790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec90106a80 0x7fec9019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec90101420 0x7fec901989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 --2- 172.21.15.78:0/418007232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec90100a80 0x7fec90198470 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 >> 172.21.15.78:0/418007232 conn(0x7fec900fc7a0 msgr2=0x7fec9006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 shutdown_connections 2024-09-16T01:10:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:32.857+0000 7fec97857640 1 -- 172.21.15.78:0/418007232 wait complete. 2024-09-16T01:10:33.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:33 smithi078 ceph-mon[30937]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:33.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:33 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/418007232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:33 smithi148 ceph-mon[35030]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:33 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/418007232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:33 smithi193 ceph-mon[35701]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-16T01:10:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:33 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/418007232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:33.565 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:33.811 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.290+0000 7fb88987a640 1 -- 172.21.15.78:0/56650220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884101de0 msgr2=0x7fb8841021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.290+0000 7fb88987a640 1 --2- 172.21.15.78:0/56650220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884101de0 0x7fb8841021c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009ca0 tx=0x7fb87c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 -- 172.21.15.78:0/56650220 shutdown_connections 2024-09-16T01:10:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 --2- 172.21.15.78:0/56650220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884106f10 0x7fb88410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 --2- 172.21.15.78:0/56650220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884102790 0x7fb8841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 --2- 172.21.15.78:0/56650220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884101de0 0x7fb8841021c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 -- 172.21.15.78:0/56650220 >> 172.21.15.78:0/56650220 conn(0x7fb8840fd8c0 msgr2=0x7fb8840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 -- 172.21.15.78:0/56650220 shutdown_connections 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 -- 172.21.15.78:0/56650220 wait complete. 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 Processor -- start 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.291+0000 7fb88987a640 1 -- start start 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 0x7fb884198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 0x7fb8841989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb884111f90 con 0x7fb884106f10 2024-09-16T01:10:34.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb884111e10 con 0x7fb884102790 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb88987a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb884112110 con 0x7fb884101de0 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35274/0 (socket says 172.21.15.78:35274) 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 -- 172.21.15.78:0/2629183294 learned_addr learned my addr 172.21.15.78:0/2629183294 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb882ffd640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 0x7fb884198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8827fc640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 0x7fb8841989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 -- 172.21.15.78:0/2629183294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 msgr2=0x7fb884198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 0x7fb884198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 -- 172.21.15.78:0/2629183294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 msgr2=0x7fb8841989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 0x7fb8841989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 -- 172.21.15.78:0/2629183294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88419d430 con 0x7fb884106f10 2024-09-16T01:10:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb882ffd640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 0x7fb884198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8827fc640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 0x7fb8841989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb8837fe640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb874009fd0 tx=0x7fb87400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.292+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874011830 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.293+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb874016070 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.293+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88419d720 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.293+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87400f2e0 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.293+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8841a5000 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.294+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8841021c0 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.295+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb8740119d0 con 0x7fb884106f10 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.296+0000 7fb888878640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 0x7fb8700a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.296+0000 7fb882ffd640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 0x7fb8700a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:34.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.296+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8740cf390 con 0x7fb884106f10 2024-09-16T01:10:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.296+0000 7fb882ffd640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 0x7fb8700a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02f540 tx=0x7fb87c0099a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.300+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb87409bd40 con 0x7fb884106f10 2024-09-16T01:10:34.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.500+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb88402d7c0 con 0x7fb884106f10 2024-09-16T01:10:34.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.501+0000 7fb888878640 1 -- 172.21.15.78:0/2629183294 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11918 (secure 0 0 0) 0x7fb874097030 con 0x7fb884106f10 2024-09-16T01:10:34.502 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:34.502 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"btime":"2024-09-16T01:09:08:102275+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:09:07.935774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:10:34.503 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-09-16T01:10:34.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 msgr2=0x7fb8700a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:34.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 0x7fb8700a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb87c02f540 tx=0x7fb87c0099a0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 msgr2=0x7fb88419cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb874009fd0 tx=0x7fb87400b8e0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 shutdown_connections 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8700a3350 0x7fb8700a5810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb884106f10 0x7fb88419cd50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884102790 0x7fb8841989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 --2- 172.21.15.78:0/2629183294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb884101de0 0x7fb884198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 >> 172.21.15.78:0/2629183294 conn(0x7fb8840fd8c0 msgr2=0x7fb8840fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 shutdown_connections 2024-09-16T01:10:34.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:34.503+0000 7fb88987a640 1 -- 172.21.15.78:0/2629183294 wait complete. 2024-09-16T01:10:35.200 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:35 smithi078 ceph-mon[30937]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:35.201 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:35 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2629183294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:35.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:10:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:35 smithi148 ceph-mon[35030]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:35 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2629183294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:35.477 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:35 smithi193 ceph-mon[35701]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-16T01:10:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:35 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2629183294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- 172.21.15.78:0/2009416380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c100e20 msgr2=0x7fb84c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 --2- 172.21.15.78:0/2009416380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c100e20 0x7fb84c101200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb840009900 tx=0x7fb84002a480 comp rx=0 tx=0).stop 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- 172.21.15.78:0/2009416380 shutdown_connections 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 --2- 172.21.15.78:0/2009416380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c102ef0 0x7fb84c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 --2- 172.21.15.78:0/2009416380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 0x7fb84c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 --2- 172.21.15.78:0/2009416380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c100e20 0x7fb84c101200 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- 172.21.15.78:0/2009416380 >> 172.21.15.78:0/2009416380 conn(0x7fb84c0fc7d0 msgr2=0x7fb84c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- 172.21.15.78:0/2009416380 shutdown_connections 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- 172.21.15.78:0/2009416380 wait complete. 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 Processor -- start 2024-09-16T01:10:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.014+0000 7fb853812640 1 -- start start 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c100e20 0x7fb84c103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 0x7fb84c103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c110f80 con 0x7fb84c102ef0 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c110e00 con 0x7fb84c100e20 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c111100 con 0x7fb84c102570 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35298/0 (socket says 172.21.15.78:35298) 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 -- 172.21.15.78:0/2028599109 learned_addr learned my addr 172.21.15.78:0/2028599109 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb850dad640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 0x7fb84c103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb8515ae640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c100e20 0x7fb84c103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 -- 172.21.15.78:0/2028599109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 msgr2=0x7fb84c103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 0x7fb84c103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 -- 172.21.15.78:0/2028599109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c100e20 msgr2=0x7fb84c103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c100e20 0x7fb84c103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 -- 172.21.15.78:0/2028599109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84c1a6a30 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb851daf640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb84400cad0 tx=0x7fb84400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84400ae20 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb844005020 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84c1a6d20 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.015+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84400f6f0 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.016+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84c1a7260 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.017+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb84c101200 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.018+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb8440051d0 con 0x7fb84c102ef0 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.019+0000 7fb83e7fc640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 0x7fb8380a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:36.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.019+0000 7fb8515ae640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 0x7fb8380a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:36.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.019+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8440d01a0 con 0x7fb84c102ef0 2024-09-16T01:10:36.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.019+0000 7fb8515ae640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 0x7fb8380a5810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb840005ab0 tx=0x7fb840005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:36.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.022+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb844014030 con 0x7fb84c102ef0 2024-09-16T01:10:36.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.191+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb84c104c80 con 0x7fb84c102ef0 2024-09-16T01:10:36.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.193+0000 7fb83e7fc640 1 -- 172.21.15.78:0/2028599109 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24645 (secure 0 0 0) 0x7fb84c104c80 con 0x7fb84c102ef0 2024-09-16T01:10:36.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:36.194 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:09:39.105086+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:10:36.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 msgr2=0x7fb8380a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:36.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 0x7fb8380a5810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb840005ab0 tx=0x7fb840005a20 comp rx=0 tx=0).stop 2024-09-16T01:10:36.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 msgr2=0x7fb84c1041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb84400cad0 tx=0x7fb84400cf90 comp rx=0 tx=0).stop 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 shutdown_connections 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fb8380a3350 0x7fb8380a5810 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c102ef0 0x7fb84c1041d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb84c102570 0x7fb84c103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 --2- 172.21.15.78:0/2028599109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb84c100e20 0x7fb84c103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 >> 172.21.15.78:0/2028599109 conn(0x7fb84c0fc7d0 msgr2=0x7fb84c06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 shutdown_connections 2024-09-16T01:10:36.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:36.195+0000 7fb853812640 1 -- 172.21.15.78:0/2028599109 wait complete. 2024-09-16T01:10:36.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:36 smithi078 ceph-mon[30937]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:36.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:36 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2028599109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:36.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:36 smithi193 ceph-mon[35701]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:36.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:36 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2028599109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:36.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:36 smithi148 ceph-mon[35030]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-16T01:10:36.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:36 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2028599109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:10:36.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T01:08:29.385380+0000', 'modified': '2024-09-16T01:09:07.935774+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24485, 'mds_1': 24460, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1399158659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1399158659}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1399158659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/1850679548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 1850679548}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 1850679548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/1637418104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1637418104}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1637418104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.148:6837/330388800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 330388800}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 330388800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.193:6839/1136630649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 1136630649}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 1136630649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6839/3106047695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 3106047695}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 3106047695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24455, 24460, 24485]} 2024-09-16T01:10:36.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T01:10:36.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-16T01:10:37.186 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 -- 172.21.15.78:0/4101047739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c100e20 msgr2=0x7fd87c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 --2- 172.21.15.78:0/4101047739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c100e20 0x7fd87c101200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd874009ca0 tx=0x7fd87402a950 comp rx=0 tx=0).stop 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 -- 172.21.15.78:0/4101047739 shutdown_connections 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 --2- 172.21.15.78:0/4101047739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c102ef0 0x7fd87c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 --2- 172.21.15.78:0/4101047739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102570 0x7fd87c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 --2- 172.21.15.78:0/4101047739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c100e20 0x7fd87c101200 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 -- 172.21.15.78:0/4101047739 >> 172.21.15.78:0/4101047739 conn(0x7fd87c0fc7d0 msgr2=0x7fd87c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 -- 172.21.15.78:0/4101047739 shutdown_connections 2024-09-16T01:10:37.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 -- 172.21.15.78:0/4101047739 wait complete. 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.672+0000 7fd8814bb640 1 Processor -- start 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 -- start start 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c100e20 0x7fd87c1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102ef0 0x7fd87c1a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c110fa0 con 0x7fd87c102570 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c110e20 con 0x7fd87c100e20 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd8814bb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c111120 con 0x7fd87c102ef0 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35314/0 (socket says 172.21.15.78:35314) 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 -- 172.21.15.78:0/1879199118 learned_addr learned my addr 172.21.15.78:0/1879199118 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87affd640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c100e20 0x7fd87c1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102ef0 0x7fd87c1a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 -- 172.21.15.78:0/1879199118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102ef0 msgr2=0x7fd87c1a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102ef0 0x7fd87c1a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 -- 172.21.15.78:0/1879199118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c100e20 msgr2=0x7fd87c1993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c100e20 0x7fd87c1993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 -- 172.21.15.78:0/1879199118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87c1a4970 con 0x7fd87c102570 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.673+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c890 tx=0x7fd86400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.674+0000 7fd873fff640 1 -- 172.21.15.78:0/1879199118 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864004220 con 0x7fd87c102570 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.674+0000 7fd873fff640 1 -- 172.21.15.78:0/1879199118 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8640043c0 con 0x7fd87c102570 2024-09-16T01:10:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.674+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87c1a4b40 con 0x7fd87c102570 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.674+0000 7fd873fff640 1 -- 172.21.15.78:0/1879199118 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640049e0 con 0x7fd87c102570 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.674+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87c1a5050 con 0x7fd87c102570 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.676+0000 7fd873fff640 1 -- 172.21.15.78:0/1879199118 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd8640079b0 con 0x7fd87c102570 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.676+0000 7fd883894740 1 Processor -- start 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.676+0000 7fd883894740 1 -- start start 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.676+0000 7fd873fff640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 0x7fd8680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 0x7fd8680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd873fff640 1 -- 172.21.15.78:0/1879199118 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8640cff70 con 0x7fd87c102570 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732d8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55d373230090 con 0x55d3732d8fc0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d3731d7a70 con 0x55d3732c98d0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd883894740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d37322c050 con 0x55d3732ca1f0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35318/0 (socket says 172.21.15.78:35318) 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 -- 172.21.15.78:0/3368306015 learned_addr learned my addr 172.21.15.78:0/3368306015 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732d8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87affd640 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 0x7fd8680a5790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00ad40 tx=0x7fd86c013650 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 -- 172.21.15.78:0/3368306015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 msgr2=0x55d3732d8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732d8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 -- 172.21.15.78:0/3368306015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 msgr2=0x55d3732c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 -- 172.21.15.78:0/3368306015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3731e4130 con 0x55d3732d8fc0 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd86c013b50 tx=0x7fd86c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8034a231fe6c52e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.677+0000 7fd8717fa640 1 -- 172.21.15.78:0/3368306015 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c026070 con 0x55d3732d8fc0 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd8717fa640 1 -- 172.21.15.78:0/3368306015 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd86c021b40 con 0x55d3732d8fc0 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd8717fa640 1 -- 172.21.15.78:0/3368306015 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00f040 con 0x55d3732d8fc0 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 -- 172.21.15.78:0/3368306015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 msgr2=0x55d3732db3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd86c013b50 tx=0x7fd86c01b000 comp rx=0 tx=0).stop 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 -- 172.21.15.78:0/3368306015 shutdown_connections 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732d8fc0 0x55d3732db3b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732d8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 --2- 172.21.15.78:0/3368306015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 -- 172.21.15.78:0/3368306015 >> 172.21.15.78:0/3368306015 conn(0x55d3732c5f60 msgr2=0x55d3732c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 -- 172.21.15.78:0/3368306015 shutdown_connections 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.678+0000 7fd883894740 1 -- 172.21.15.78:0/3368306015 wait complete. 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 -- start start 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 Processor -- start 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 0x55d3732d88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3732d6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55d3732dbb90 con 0x55d3732ca1f0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d3731eaf20 con 0x55d3732d8fc0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3732dbd10 con 0x55d3732c98d0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd883894740 10 client.0 _ll_register_callbacks cb 0x55d3731b4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35334/0 (socket says 172.21.15.78:35334) 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 -- 172.21.15.78:0/915485531 learned_addr learned my addr 172.21.15.78:0/915485531 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3732d6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87affd640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 0x55d3732d88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 -- 172.21.15.78:0/915485531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 msgr2=0x55d3732d88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 0x55d3732d88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 -- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 msgr2=0x55d3732d6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3732d6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87a7fc640 1 -- 172.21.15.78:0/915485531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d372fa12b0 con 0x55d3732ca1f0 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3732d6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.679+0000 7fd87affd640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 0x55d3732d88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd86409c000 tx=0x7fd86409c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4165534840e9347d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86401d040 con 0x55d3732ca1f0 2024-09-16T01:10:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8640a00d0 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd86c021b40 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd8640a4ce0 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 10 client.15162 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d3732e75e0 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd86c021b40 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fd8640cdb70 con 0x55d3732ca1f0 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 10 client.15162 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 10 client.15162 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 20 client.15162 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d3732e9ba0 0x55d3732eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 4 client.15162 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.680+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d37323d600 con 0x55d3732e9ba0 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.681+0000 7fd87affd640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d3732e9ba0 0x55d3732eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.681+0000 7fd87affd640 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d3732e9ba0 0x55d3732eb010 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af63b6933942c096 server_cookie=d2b99658619827b9 in_seq=0 out_seq=0 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.681+0000 7fd8627fc640 10 client.15162 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd8627fc640 1 -- 172.21.15.78:0/915485531 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d37323d600 con 0x55d3732e9ba0 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd8627fc640 10 client.15162 handle_command_reply: tid=0 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 client.15162 shutdown 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 msgr2=0x55d3732d5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd86409c000 tx=0x7fd86409c830 comp rx=0 tx=0).stop 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 shutdown_connections 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d3732e9ba0 0x55d3732eb010 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3732d6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd8627fc640 0 client.15162 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5c40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 --2- 172.21.15.78:0/915485531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732c98d0 0x55d3732d88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 >> 172.21.15.78:0/915485531 conn(0x55d3732c5f60 msgr2=0x55d3732cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 shutdown_connections 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 1 -- 172.21.15.78:0/915485531 wait complete. 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.682+0000 7fd883894740 20 client.15162 trim_cache size 0 max 16384 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 Processor -- start 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 -- start start 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732d8fc0 0x55d3732d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55d3732e9350 con 0x55d3732ca1f0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d3732e9a20 con 0x55d3732c98d0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd883894740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3732ea0f0 con 0x55d3732d8fc0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd87a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35348/0 (socket says 172.21.15.78:35348) 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.700+0000 7fd87a7fc640 1 -- 172.21.15.78:0/959447746 learned_addr learned my addr 172.21.15.78:0/959447746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732d8fc0 0x55d3732d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 -- 172.21.15.78:0/959447746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732d8fc0 msgr2=0x55d3732d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87affd640 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732d8fc0 0x55d3732d4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 -- 172.21.15.78:0/959447746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 msgr2=0x55d3732eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732eb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 -- 172.21.15.78:0/959447746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3732d6c80 con 0x55d3732ca1f0 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd8640058c0 tx=0x7fd86409b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba23c08919ded05f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd870ff9640 1 -- 172.21.15.78:0/959447746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86401d030 con 0x55d3732ca1f0 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd870ff9640 1 -- 172.21.15.78:0/959447746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8640b5060 con 0x55d3732ca1f0 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.701+0000 7fd870ff9640 1 -- 172.21.15.78:0/959447746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640a4ab0 con 0x55d3732ca1f0 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- 172.21.15.78:0/959447746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 msgr2=0x55d3732d5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd8640058c0 tx=0x7fd86409b9d0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- 172.21.15.78:0/959447746 shutdown_connections 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732d8fc0 0x55d3732d4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732ca1f0 0x55d3732d5500 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- 172.21.15.78:0/959447746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732c98d0 0x55d3732eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- 172.21.15.78:0/959447746 >> 172.21.15.78:0/959447746 conn(0x55d3732c5f60 msgr2=0x55d3732c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- 172.21.15.78:0/959447746 shutdown_connections 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- 172.21.15.78:0/959447746 wait complete. 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- start start 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 Processor -- start 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732eccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3733522a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55d3732eae90 con 0x55d3732c98d0 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d3732ea7c0 con 0x55d3732d8fc0 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d3732eb560 con 0x55d3732ca1f0 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd883894740 10 client.0 _ll_register_callbacks cb 0x7fd86400a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd87affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd87affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35358/0 (socket says 172.21.15.78:35358) 2024-09-16T01:10:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.702+0000 7fd87affd640 1 -- 172.21.15.78:0/1480851434 learned_addr learned my addr 172.21.15.78:0/1480851434 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732eccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87b7fe640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3733522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 -- 172.21.15.78:0/1480851434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 msgr2=0x55d3732eccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732eccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 -- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 msgr2=0x55d3733522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3733522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 -- 172.21.15.78:0/1480851434 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3733527e0 con 0x55d3732c98d0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd87affd640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd874006010 tx=0x7fd87402fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2154705cd7e875b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd8627fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874034040 con 0x55d3732c98d0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd87402f5a0 con 0x55d3732c98d0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8640b5060 con 0x55d3732c98d0 2024-09-16T01:10:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd874053050 con 0x55d3732c98d0 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd883894740 10 client.15168 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d3733539a0 con 0x55d3732c98d0 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.703+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8640b5060 con 0x55d3732c98d0 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fd874004530 con 0x55d3732c98d0 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 10 client.15168 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 10 client.15168 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 20 client.15168 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d373358590 0x55d3733572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 4 client.15168 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "status"}) -- 0x55d373237b10 con 0x55d373358590 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d373358590 0x55d3733572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd8627fc640 10 client.15168 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.704+0000 7fd87a7fc640 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d373358590 0x55d3733572a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f7e368870646162 server_cookie=7c6759e8d8e40034 in_seq=0 out_seq=0 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8627fc640 1 -- 172.21.15.78:0/1480851434 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x55d373237b10 con 0x55d373358590 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8627fc640 10 client.15168 handle_command_reply: tid=0 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 client.15168 shutdown 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 msgr2=0x55d3732ec7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd874006010 tx=0x7fd87402fa90 comp rx=0 tx=0).stop 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 shutdown_connections 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x55d373358590 0x55d3733572a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.727 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24455, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 92.61109356, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 151.018107461 2024-09-16T01:10:37.728 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d3732d8fc0 0x55d3733522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8627fc640 0 client.15168 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d3732ca1f0 0x55d3732eccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 --2- 172.21.15.78:0/1480851434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d3732c98d0 0x55d3732ec7b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 >> 172.21.15.78:0/1480851434 conn(0x55d3732c5f60 msgr2=0x55d3732d5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 shutdown_connections 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 1 -- 172.21.15.78:0/1480851434 wait complete. 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd883894740 20 client.15168 trim_cache size 0 max 16384 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 msgr2=0x7fd8680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 0x7fd8680a5790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00ad40 tx=0x7fd86c013650 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.705+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 msgr2=0x7fd87c1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd86400c890 tx=0x7fd86400cd50 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 shutdown_connections 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fd8680a32d0 0x7fd8680a5790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd87c102ef0 0x7fd87c1a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd87c102570 0x7fd87c1998f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 --2- 172.21.15.78:0/1879199118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd87c100e20 0x7fd87c1993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 >> 172.21.15.78:0/1879199118 conn(0x7fd87c0fc7d0 msgr2=0x7fd87c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 shutdown_connections 2024-09-16T01:10:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:37.706+0000 7fd8814bb640 1 -- 172.21.15.78:0/1879199118 wait complete. 2024-09-16T01:10:38.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T01:10:38.805 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:39 smithi078 ceph-mon[30937]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-16T01:10:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-09-16T01:10:39.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:39 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 -- 172.21.15.78:0/3179664473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8106a80 msgr2=0x7f79e8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 --2- 172.21.15.78:0/3179664473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8106a80 0x7f79e8106e60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009cd0 tx=0x7f79e002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 -- 172.21.15.78:0/3179664473 shutdown_connections 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 --2- 172.21.15.78:0/3179664473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8101420 0x7f79e810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 --2- 172.21.15.78:0/3179664473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 0x7f79e8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 --2- 172.21.15.78:0/3179664473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8106a80 0x7f79e8106e60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 -- 172.21.15.78:0/3179664473 >> 172.21.15.78:0/3179664473 conn(0x7f79e80fc7a0 msgr2=0x7f79e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 -- 172.21.15.78:0/3179664473 shutdown_connections 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.306+0000 7f79ecaff640 1 -- 172.21.15.78:0/3179664473 wait complete. 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 Processor -- start 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 -- start start 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 0x7f79e80fce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 0x7f79e80fd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8110e80 con 0x7f79e8101420 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8110d00 con 0x7f79e8106a80 2024-09-16T01:10:39.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79ecaff640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8111000 con 0x7f79e8100a80 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35374/0 (socket says 172.21.15.78:35374) 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79e5d9b640 1 -- 172.21.15.78:0/604746398 learned_addr learned my addr 172.21.15.78:0/604746398 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.307+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 0x7f79e80fd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e659c640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 0x7f79e80fce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 -- 172.21.15.78:0/604746398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 msgr2=0x7f79e80fce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 0x7f79e80fce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 -- 172.21.15.78:0/604746398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 msgr2=0x7f79e80fd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 0x7f79e80fd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 -- 172.21.15.78:0/604746398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e81a6960 con 0x7f79e8101420 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 0x7f79e80fd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:39.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f79d000d770 tx=0x7f79d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79df7fe640 1 -- 172.21.15.78:0/604746398 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0017070 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e81a6c50 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79df7fe640 1 -- 172.21.15.78:0/604746398 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d0005500 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79df7fe640 1 -- 172.21.15.78:0/604746398 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0004e00 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.308+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e81a7190 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79df7fe640 1 -- 172.21.15.78:0/604746398 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f79d0004fa0 con 0x7f79e8101420 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 Processor -- start 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 -- start start 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec979d0 0x5564dec97db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564deca2bb0 0x5564deca4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564debfe090 con 0x5564dec982f0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5564deba5a70 con 0x5564dec979d0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.311+0000 7f79eeed8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564debfa050 con 0x5564deca2bb0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35384/0 (socket says 172.21.15.78:35384) 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 -- 172.21.15.78:0/2261638847 learned_addr learned my addr 172.21.15.78:0/2261638847 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564deca2bb0 0x5564deca4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec979d0 0x5564dec97db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79df7fe640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 0x7f79d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 -- 172.21.15.78:0/2261638847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564deca2bb0 msgr2=0x5564deca4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564deca2bb0 0x5564deca4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 -- 172.21.15.78:0/2261638847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec979d0 msgr2=0x5564dec97db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec979d0 0x5564dec97db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 0x7f79d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 -- 172.21.15.78:0/2261638847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564debb2130 con 0x5564dec982f0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79df7fe640 1 -- 172.21.15.78:0/604746398 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f79d00cf7a0 con 0x7f79e8101420 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e659c640 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000f80 tx=0x7f79e0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be68c03aa4bfa6f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 0x7f79d40a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f79d80136c0 tx=0x7f79d8004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79dcff9640 1 -- 172.21.15.78:0/2261638847 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00318c0 con 0x5564dec982f0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79dcff9640 1 -- 172.21.15.78:0/2261638847 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0036040 con 0x5564dec982f0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.312+0000 7f79dcff9640 1 -- 172.21.15.78:0/2261638847 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0004560 con 0x5564dec982f0 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 -- 172.21.15.78:0/2261638847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 msgr2=0x5564deca2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000f80 tx=0x7f79e0009a20 comp rx=0 tx=0).stop 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 -- 172.21.15.78:0/2261638847 shutdown_connections 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564deca2bb0 0x5564deca4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564dec982f0 0x5564deca2670 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 --2- 172.21.15.78:0/2261638847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec979d0 0x5564dec97db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 -- 172.21.15.78:0/2261638847 >> 172.21.15.78:0/2261638847 conn(0x5564dec93f60 msgr2=0x5564dec94340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 -- 172.21.15.78:0/2261638847 shutdown_connections 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.313+0000 7f79eeed8740 1 -- 172.21.15.78:0/2261638847 wait complete. 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 -- start start 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 Processor -- start 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564deca2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564deca5780 con 0x5564deca2bb0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5564debb8f20 con 0x5564dec982f0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564deca5900 con 0x5564dec979d0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79eeed8740 10 client.0 _ll_register_callbacks cb 0x5564deb82030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35398/0 (socket says 172.21.15.78:35398) 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 -- 172.21.15.78:0/879228117 learned_addr learned my addr 172.21.15.78:0/879228117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e659c640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564deca2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 -- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 msgr2=0x5564deca2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564deca2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 -- 172.21.15.78:0/879228117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 msgr2=0x5564dec9f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.314+0000 7f79e6d9d640 1 -- 172.21.15.78:0/879228117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564de96f2b0 con 0x5564deca2bb0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79d80117f0 tx=0x7f79d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a02f34407578095 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8019070 con 0x5564deca2bb0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f79d8010de0 con 0x5564deca2bb0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e0036040 con 0x5564deca2bb0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f79d8011e30 con 0x5564deca2bb0 2024-09-16T01:10:39.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 10 client.15180 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5564decb1100 con 0x5564deca2bb0 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e0036040 con 0x5564deca2bb0 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f79d8020490 con 0x5564deca2bb0 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 10 client.15180 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 10 client.15180 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.315+0000 7f79eeed8740 20 client.15180 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564decb3730 0x5564decb4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79eeed8740 4 client.15180 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564dec0b600 con 0x5564decb3730 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79e659c640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564decb3730 0x5564decb4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79cdffb640 10 client.15180 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.316+0000 7f79e659c640 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564decb3730 0x5564decb4ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=652776b86f2cfb6 server_cookie=b67d34eed24a6277 in_seq=0 out_seq=0 2024-09-16T01:10:39.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79cdffb640 1 -- 172.21.15.78:0/879228117 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5564dec0b600 con 0x5564decb3730 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79cdffb640 10 client.15180 handle_command_reply: tid=0 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 client.15180 shutdown 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 msgr2=0x5564dec9fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79d80117f0 tx=0x7f79d80046c0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 shutdown_connections 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564decb3730 0x5564decb4ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79cdffb640 0 client.15180 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 --2- 172.21.15.78:0/879228117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564deca2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 >> 172.21.15.78:0/879228117 conn(0x5564dec93f60 msgr2=0x5564dec99c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 shutdown_connections 2024-09-16T01:10:39.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 1 -- 172.21.15.78:0/879228117 wait complete. 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.317+0000 7f79eeed8740 20 client.15180 trim_cache size 0 max 16384 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 Processor -- start 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 -- start start 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564decb2ee0 con 0x5564deca2bb0 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5564decb35b0 con 0x5564dec982f0 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79eeed8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564decb3c80 con 0x5564dec979d0 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35406/0 (socket says 172.21.15.78:35406) 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 -- 172.21.15.78:0/107895722 learned_addr learned my addr 172.21.15.78:0/107895722 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:39.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e659c640 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 -- 172.21.15.78:0/107895722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 msgr2=0x5564decb4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 -- 172.21.15.78:0/107895722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 msgr2=0x5564dec9f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.336+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79e6d9d640 1 -- 172.21.15.78:0/107895722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564dec93d90 con 0x5564deca2bb0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79d8000c00 tx=0x7f79d8013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d121de68cbbe8b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79cffff640 1 -- 172.21.15.78:0/107895722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8039070 con 0x5564deca2bb0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79cffff640 1 -- 172.21.15.78:0/107895722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d8010820 con 0x5564deca2bb0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79cffff640 1 -- 172.21.15.78:0/107895722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8019040 con 0x5564deca2bb0 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79eeed8740 1 -- 172.21.15.78:0/107895722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 msgr2=0x5564dec9fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.337+0000 7f79eeed8740 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79d8000c00 tx=0x7f79d8013f50 comp rx=0 tx=0).stop 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- 172.21.15.78:0/107895722 shutdown_connections 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564dec9fc50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564dec9f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- 172.21.15.78:0/107895722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- 172.21.15.78:0/107895722 >> 172.21.15.78:0/107895722 conn(0x5564dec93f60 msgr2=0x5564dec9ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- 172.21.15.78:0/107895722 shutdown_connections 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- 172.21.15.78:0/107895722 wait complete. 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- start start 2024-09-16T01:10:39.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 Processor -- start 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564decb8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5564decb4a20 con 0x5564deca2bb0 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5564decb4350 con 0x5564dec982f0 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564decb50f0 con 0x5564dec979d0 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.338+0000 7f79eeed8740 10 client.0 _ll_register_callbacks cb 0x7f79d800bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:39.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35418/0 (socket says 172.21.15.78:35418) 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 -- 172.21.15.78:0/3179241907 learned_addr learned my addr 172.21.15.78:0/3179241907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e659c640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e5d9b640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564decb8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 -- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 msgr2=0x5564decb7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 -- 172.21.15.78:0/3179241907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 msgr2=0x5564decb8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564decb8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 -- 172.21.15.78:0/3179241907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564ded1c370 con 0x5564deca2bb0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79e6d9d640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f79d8011fd0 tx=0x7f79d8020520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a810f21943720a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8041070 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9179+0+0 (secure 0 0 0) 0x7f79d8010de0 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79d8010820 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f79d8039070 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79eeed8740 10 client.15186 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.339+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5564ded1d4b0 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79d8010820 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f79d8035af0 con 0x5564deca2bb0 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 10 client.15186 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 10 client.15186 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 20 client.15186 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:39.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564ded21ca0 0x5564ded209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 4 client.15186 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "status"}) -- 0x5564dec05b10 con 0x5564ded21ca0 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79e659c640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564ded21ca0 0x5564ded209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79e659c640 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564ded21ca0 0x5564ded209b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cbec016636f2fb25 server_cookie=bc2afe10e472f21e in_seq=0 out_seq=0 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.340+0000 7f79cdffb640 10 client.15186 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79cdffb640 1 -- 172.21.15.78:0/3179241907 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5564dec05b10 con 0x5564ded21ca0 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79cdffb640 10 client.15186 handle_command_reply: tid=0 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 client.15186 shutdown 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 msgr2=0x5564ded1be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f79d8011fd0 tx=0x7f79d8020520 comp rx=0 tx=0).stop 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 shutdown_connections 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x5564ded21ca0 0x5564ded209b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564deca2bb0 0x5564ded1be30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5564dec982f0 0x5564decb8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 --2- 172.21.15.78:0/3179241907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564dec979d0 0x5564decb7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 >> 172.21.15.78:0/3179241907 conn(0x5564dec93f60 msgr2=0x5564dec9f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79cdffb640 0 client.15186 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 shutdown_connections 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 1 -- 172.21.15.78:0/3179241907 wait complete. 2024-09-16T01:10:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79eeed8740 20 client.15186 trim_cache size 0 max 16384 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24460, 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 95.255970297000005, 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 92, 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 150.12533077000001 2024-09-16T01:10:39.364 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 msgr2=0x7f79d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 0x7f79d40a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f79d80136c0 tx=0x7f79d8004730 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 msgr2=0x7f79e80fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.341+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f79d000d770 tx=0x7f79d000dc30 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 shutdown_connections 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f79d40a3240 0x7f79d40a5700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79e8106a80 0x7f79e80fd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e8101420 0x7f79e80fd340 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 --2- 172.21.15.78:0/604746398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79e8100a80 0x7f79e80fce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 >> 172.21.15.78:0/604746398 conn(0x7f79e80fc7a0 msgr2=0x7f79e806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 shutdown_connections 2024-09-16T01:10:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:39.342+0000 7f79ecaff640 1 -- 172.21.15.78:0/604746398 wait complete. 2024-09-16T01:10:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:39 smithi148 ceph-mon[35030]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-16T01:10:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-09-16T01:10:39.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:39 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-09-16T01:10:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:39 smithi193 ceph-mon[35701]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-09-16T01:10:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-16T01:10:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-09-16T01:10:39.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:39 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-09-16T01:10:40.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-09-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:40 smithi078 ceph-mon[30937]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:40 smithi078 ceph-mon[30937]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T01:10:40.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:40 smithi078 ceph-mon[30937]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T01:10:40.303 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:40.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-16T01:10:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-09-16T01:10:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:40 smithi148 ceph-mon[35030]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-16T01:10:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:40 smithi148 ceph-mon[35030]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T01:10:40.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:40 smithi148 ceph-mon[35030]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T01:10:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-16T01:10:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-09-16T01:10:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:40 smithi193 ceph-mon[35701]: from='mgr.14150 172.21.15.78:0/453231060' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-09-16T01:10:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:40 smithi193 ceph-mon[35701]: osdmap e92: 12 total, 12 up, 12 in 2024-09-16T01:10:40.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:40 smithi193 ceph-mon[35701]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-16T01:10:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 -- 172.21.15.78:0/2773278234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28105a70 msgr2=0x7f0a28105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 --2- 172.21.15.78:0/2773278234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28105a70 0x7f0a28105e50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009900 tx=0x7f0a2402a410 comp rx=0 tx=0).stop 2024-09-16T01:10:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 -- 172.21.15.78:0/2773278234 shutdown_connections 2024-09-16T01:10:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 --2- 172.21.15.78:0/2773278234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 0x7f0a2810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 --2- 172.21.15.78:0/2773278234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28101220 0x7f0a28101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 --2- 172.21.15.78:0/2773278234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28105a70 0x7f0a28105e50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 -- 172.21.15.78:0/2773278234 >> 172.21.15.78:0/2773278234 conn(0x7f0a280fbee0 msgr2=0x7f0a280fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 -- 172.21.15.78:0/2773278234 shutdown_connections 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.805+0000 7f0a305a1640 1 -- 172.21.15.78:0/2773278234 wait complete. 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 Processor -- start 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 -- start start 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 0x7f0a28197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28105a70 0x7f0a2819bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28110f50 con 0x7f0a28101220 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28110dd0 con 0x7f0a28101ba0 2024-09-16T01:10:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a305a1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281110d0 con 0x7f0a28105a70 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35428/0 (socket says 172.21.15.78:35428) 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.806+0000 7f0a2e33d640 1 -- 172.21.15.78:0/438007043 learned_addr learned my addr 172.21.15.78:0/438007043 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 0x7f0a28197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28105a70 0x7f0a2819bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 -- 172.21.15.78:0/438007043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28105a70 msgr2=0x7f0a2819bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28105a70 0x7f0a2819bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 -- 172.21.15.78:0/438007043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 msgr2=0x7f0a28197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 0x7f0a28197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 -- 172.21.15.78:0/438007043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2819c3a0 con 0x7f0a28101220 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0a24006010 tx=0x7f0a24003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a1f7fe640 1 -- 172.21.15.78:0/438007043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2403f070 con 0x7f0a28101220 2024-09-16T01:10:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2819c630 con 0x7f0a28101220 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a1f7fe640 1 -- 172.21.15.78:0/438007043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a24031c30 con 0x7f0a28101220 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a1f7fe640 1 -- 172.21.15.78:0/438007043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24035030 con 0x7f0a28101220 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.807+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281a3f70 con 0x7f0a28101220 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.809+0000 7f0a1f7fe640 1 -- 172.21.15.78:0/438007043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0a24042070 con 0x7f0a28101220 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 Processor -- start 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 -- start start 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 0x558d427e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 0x558d427f3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d42748090 con 0x558d427e4000 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d426efa70 con 0x558d427f0c70 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a3297a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d42744050 con 0x558d427e5230 2024-09-16T01:10:40.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a1f7fe640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 0x7f0a140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 0x7f0a140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35430/0 (socket says 172.21.15.78:35430) 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a2e33d640 1 -- 172.21.15.78:0/669167780 learned_addr learned my addr 172.21.15.78:0/669167780 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.810+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 0x558d427f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a1f7fe640 1 -- 172.21.15.78:0/438007043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0a240f0470 con 0x7f0a28101220 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 0x558d427e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 -- 172.21.15.78:0/669167780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 msgr2=0x558d427e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 0x558d427e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 -- 172.21.15.78:0/669167780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 msgr2=0x558d427f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 0x558d427f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 -- 172.21.15.78:0/669167780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d426fc130 con 0x558d427e4000 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 0x558d427f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 0x558d427e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 0x7f0a140a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009fd0 tx=0x7f0a20012af0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0a240c4e20 tx=0x7f0a2408f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183b7bb5717ffd79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a1cff9640 1 -- 172.21.15.78:0/669167780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24049030 con 0x558d427e4000 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a1cff9640 1 -- 172.21.15.78:0/669167780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a24031de0 con 0x558d427e4000 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.811+0000 7f0a1cff9640 1 -- 172.21.15.78:0/669167780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240c5350 con 0x558d427e4000 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- 172.21.15.78:0/669167780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 msgr2=0x558d427e4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0a240c4e20 tx=0x7f0a2408f6d0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- 172.21.15.78:0/669167780 shutdown_connections 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427f0c70 0x558d427f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427e4460 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 --2- 172.21.15.78:0/669167780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427e5230 0x558d427e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- 172.21.15.78:0/669167780 >> 172.21.15.78:0/669167780 conn(0x558d427e0170 msgr2=0x558d427e0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- 172.21.15.78:0/669167780 shutdown_connections 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- 172.21.15.78:0/669167780 wait complete. 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 -- start start 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.812+0000 7f0a3297a740 1 Processor -- start 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e4000 0x558d427f0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d427edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d427f3940 con 0x558d427e5230 2024-09-16T01:10:40.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d42702f20 con 0x558d427e4000 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d427f3ac0 con 0x558d427f0c70 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 10 client.0 _ll_register_callbacks cb 0x558d426cc030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35438/0 (socket says 172.21.15.78:35438) 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 -- 172.21.15.78:0/2692578980 learned_addr learned my addr 172.21.15.78:0/2692578980 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e4000 0x558d427f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d427edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 -- 172.21.15.78:0/2692578980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 msgr2=0x558d427edf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d427edf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 -- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e4000 msgr2=0x558d427f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e4000 0x558d427f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 -- 172.21.15.78:0/2692578980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d424b92b0 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d427edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800b9d0 tx=0x7f0a1800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891498dd2a2f5d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18005020 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0a180051c0 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a24031de0 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a180053c0 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 10 client.15198 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.813+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558d427ff350 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a24031de0 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a1801b630 con 0x558d427e5230 2024-09-16T01:10:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 10 client.15198 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 10 client.15198 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 20 client.15198 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d428019a0 0x558d42802e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 4 client.15198 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d42755600 con 0x558d428019a0 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.814+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d428019a0 0x558d42802e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d428019a0 0x558d42802e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a963f16f9b7293ac server_cookie=6db6b090332a7268 in_seq=0 out_seq=0 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a11ffb640 10 client.15198 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a11ffb640 1 -- 172.21.15.78:0/2692578980 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d42755600 con 0x558d428019a0 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a11ffb640 10 client.15198 handle_command_reply: tid=0 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 client.15198 shutdown 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 msgr2=0x558d427ed990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800b9d0 tx=0x7f0a1800be90 comp rx=0 tx=0).stop 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 shutdown_connections 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d428019a0 0x558d42802e10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d427edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e5230 0x558d427ed990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2692578980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e4000 0x558d427f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a11ffb640 0 client.15198 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.815+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 >> 172.21.15.78:0/2692578980 conn(0x558d427e0170 msgr2=0x558d427e5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.816+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 shutdown_connections 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.816+0000 7f0a3297a740 1 -- 172.21.15.78:0/2692578980 wait complete. 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.816+0000 7f0a3297a740 20 client.15198 trim_cache size 0 max 16384 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 Processor -- start 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 -- start start 2024-09-16T01:10:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d427ed790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d42801150 con 0x558d427e4000 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d42801820 con 0x558d427e5230 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a3297a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d42801ef0 con 0x558d427f0c70 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35450/0 (socket says 172.21.15.78:35450) 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 -- 172.21.15.78:0/2304144292 learned_addr learned my addr 172.21.15.78:0/2304144292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 -- 172.21.15.78:0/2304144292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 msgr2=0x558d4280b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 -- 172.21.15.78:0/2304144292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 msgr2=0x558d427ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d427ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 -- 172.21.15.78:0/2304144292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d427dcde0 con 0x558d427e4000 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.833+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0a240b9160 tx=0x7f0a2403b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f66057fa863b15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a13fff640 1 -- 172.21.15.78:0/2304144292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240d4070 con 0x558d427e4000 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a13fff640 1 -- 172.21.15.78:0/2304144292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a240c5350 con 0x558d427e4000 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a13fff640 1 -- 172.21.15.78:0/2304144292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24049040 con 0x558d427e4000 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- 172.21.15.78:0/2304144292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 msgr2=0x558d427ed250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0a240b9160 tx=0x7f0a2403b870 comp rx=0 tx=0).stop 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- 172.21.15.78:0/2304144292 shutdown_connections 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d427ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 --2- 172.21.15.78:0/2304144292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d427ed250 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- 172.21.15.78:0/2304144292 >> 172.21.15.78:0/2304144292 conn(0x558d427e0170 msgr2=0x558d427ec260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- 172.21.15.78:0/2304144292 shutdown_connections 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- 172.21.15.78:0/2304144292 wait complete. 2024-09-16T01:10:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.834+0000 7f0a3297a740 1 -- start start 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 Processor -- start 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d4280b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x558d42802c90 con 0x558d427e4000 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d428025c0 con 0x558d427e5230 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d42803360 con 0x558d427f0c70 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a3297a740 10 client.0 _ll_register_callbacks cb 0x7f0a180042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35466/0 (socket says 172.21.15.78:35466) 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 -- 172.21.15.78:0/1255104337 learned_addr learned my addr 172.21.15.78:0/1255104337 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d4280b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2eb3e640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 -- 172.21.15.78:0/1255104337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 msgr2=0x558d4280b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 -- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 msgr2=0x558d4280b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d4280b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 -- 172.21.15.78:0/1255104337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d428061b0 con 0x558d427e4000 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d4280b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a2e33d640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a24005ab0 tx=0x7f0a240f02e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb29b68252981fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.835+0000 7f0a11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24049070 con 0x558d427e4000 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0a240b8e00 con 0x558d427e4000 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a240c5350 con 0x558d427e4000 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a240d4040 con 0x558d427e4000 2024-09-16T01:10:40.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 10 client.15204 fetch_fsmap learned FSMap version 24 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558d4286c5c0 con 0x558d427e4000 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a240c5350 con 0x558d427e4000 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a2403b460 con 0x558d427e4000 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 10 client.15204 fetch_fsmap finished waiting for FSMap version 24 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 10 client.15204 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 20 client.15204 populate_metadata read hostname 'smithi078' 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d42870f30 0x558d4286fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 4 client.15204 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "status"}) -- 0x558d4274fb10 con 0x558d42870f30 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.836+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d42870f30 0x558d4286fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a2db3c640 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d42870f30 0x558d4286fc40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b8e03cde3014a87 server_cookie=3f7c8eeb9724d8a4 in_seq=0 out_seq=0 2024-09-16T01:10:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a11ffb640 10 client.15204 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a11ffb640 1 -- 172.21.15.78:0/1255104337 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558d4274fb10 con 0x558d42870f30 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a11ffb640 10 client.15204 handle_command_reply: tid=0 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 client.15204 shutdown 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 msgr2=0x558d4280af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a24005ab0 tx=0x7f0a240f02e0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 shutdown_connections 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558d42870f30 0x558d4286fc40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d427f0c70 0x558d4280b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d427e5230 0x558d4280b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 --2- 172.21.15.78:0/1255104337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558d427e4000 0x558d4280af60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 >> 172.21.15.78:0/1255104337 conn(0x558d427e0170 msgr2=0x558d427ed250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a11ffb640 0 client.15204 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 shutdown_connections 2024-09-16T01:10:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 1 -- 172.21.15.78:0/1255104337 wait complete. 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.837+0000 7f0a3297a740 20 client.15204 trim_cache size 0 max 16384 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:10:40.856 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 131.10639428100001, 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 93, 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 143.06486859500001 2024-09-16T01:10:40.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 msgr2=0x7f0a140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 0x7f0a140a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009fd0 tx=0x7f0a20012af0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 msgr2=0x7f0a281973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0a24006010 tx=0x7f0a24003c10 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 shutdown_connections 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f0a140a3240 0x7f0a140a5700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28105a70 0x7f0a2819bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a28101ba0 0x7f0a28197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 --2- 172.21.15.78:0/438007043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a28101220 0x7f0a281973f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 >> 172.21.15.78:0/438007043 conn(0x7f0a280fbee0 msgr2=0x7f0a280fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:40.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 shutdown_connections 2024-09-16T01:10:40.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:40.838+0000 7f0a305a1640 1 -- 172.21.15.78:0/438007043 wait complete. 2024-09-16T01:10:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:41 smithi148 ceph-mon[35030]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-09-16T01:10:41.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:41 smithi148 ceph-mon[35030]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T01:10:41.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:41 smithi078 ceph-mon[30937]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-09-16T01:10:41.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:41 smithi078 ceph-mon[30937]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T01:10:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:41 smithi193 ceph-mon[35701]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 307 B/s wr, 6 op/s 2024-09-16T01:10:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:41 smithi193 ceph-mon[35701]: osdmap e93: 12 total, 12 up, 12 in 2024-09-16T01:10:41.613 INFO:teuthology.run_tasks:Running task sequential... 2024-09-16T01:10:41.625 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-16T01:10:41.625 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-16T01:10:41.626 INFO:teuthology.task.print:Enabling mgr modules 2024-09-16T01:10:41.626 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-09-16T01:10:41.627 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T01:10:41.627 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:41.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-09-16T01:10:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 -- 172.21.15.78:0/2283891584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90101c40 msgr2=0x7f6b901020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 --2- 172.21.15.78:0/2283891584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90101c40 0x7f6b901020c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009900 tx=0x7f6b8402a410 comp rx=0 tx=0).stop 2024-09-16T01:10:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 -- 172.21.15.78:0/2283891584 shutdown_connections 2024-09-16T01:10:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 --2- 172.21.15.78:0/2283891584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90102600 0x7f6b90108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 --2- 172.21.15.78:0/2283891584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90101c40 0x7f6b901020c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 --2- 172.21.15.78:0/2283891584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90100a50 0x7f6b90100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 -- 172.21.15.78:0/2283891584 >> 172.21.15.78:0/2283891584 conn(0x7f6b900fc1b0 msgr2=0x7f6b900fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 -- 172.21.15.78:0/2283891584 shutdown_connections 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.728+0000 7f6b986d0640 1 -- 172.21.15.78:0/2283891584 wait complete. 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 Processor -- start 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 -- start start 2024-09-16T01:10:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 0x7f6b90196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90101c40 0x7f6b90197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c6d0 con 0x7f6b90102600 2024-09-16T01:10:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c550 con 0x7f6b90100a50 2024-09-16T01:10:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b986d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c850 con 0x7f6b90101c40 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35474/0 (socket says 172.21.15.78:35474) 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 -- 172.21.15.78:0/1927597518 learned_addr learned my addr 172.21.15.78:0/1927597518 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b9646c640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 0x7f6b90196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b95c6b640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90101c40 0x7f6b90197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 -- 172.21.15.78:0/1927597518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90101c40 msgr2=0x7f6b90197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90101c40 0x7f6b90197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 -- 172.21.15.78:0/1927597518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 msgr2=0x7f6b90196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 0x7f6b90196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b96c6d640 1 -- 172.21.15.78:0/1927597518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9019e900 con 0x7f6b90102600 2024-09-16T01:10:41.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.729+0000 7f6b9646c640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 0x7f6b90196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b96c6d640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009fd0 tx=0x7f6b8000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80017070 con 0x7f6b90102600 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9019ebf0 con 0x7f6b90102600 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b80003e50 con 0x7f6b90102600 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b800106c0 con 0x7f6b90102600 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.730+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9019f140 con 0x7f6b90102600 2024-09-16T01:10:41.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.731+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9019f310 con 0x7f6b90102600 2024-09-16T01:10:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.732+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6b80003970 con 0x7f6b90102600 2024-09-16T01:10:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.733+0000 7f6b8f7fe640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 0x7f6b780a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.733+0000 7f6b9646c640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 0x7f6b780a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:41.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.733+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f6b800cfb10 con 0x7f6b90102600 2024-09-16T01:10:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.733+0000 7f6b9646c640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 0x7f6b780a57b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6b880058c0 tx=0x7f6b88005d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:41.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.736+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6b8009c480 con 0x7f6b90102600 2024-09-16T01:10:41.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:41.948+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f6b901077b0 con 0x7f6b90102600 2024-09-16T01:10:42.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.099+0000 7f6b8f7fe640 1 -- 172.21.15.78:0/1927597518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f6b8009bd80 con 0x7f6b90102600 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 msgr2=0x7f6b780a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 0x7f6b780a57b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6b880058c0 tx=0x7f6b88005d70 comp rx=0 tx=0).stop 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 msgr2=0x7f6b9019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009fd0 tx=0x7f6b8000b260 comp rx=0 tx=0).stop 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 shutdown_connections 2024-09-16T01:10:42.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f6b780a32f0 0x7f6b780a57b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b90102600 0x7f6b9019e260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b90101c40 0x7f6b90197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 --2- 172.21.15.78:0/1927597518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b90100a50 0x7f6b90196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.102+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 >> 172.21.15.78:0/1927597518 conn(0x7f6b900fc1b0 msgr2=0x7f6b900fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.103+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 shutdown_connections 2024-09-16T01:10:42.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:42.103+0000 7f6b986d0640 1 -- 172.21.15.78:0/1927597518 wait complete. 2024-09-16T01:10:42.119 INFO:teuthology.run_tasks:Running task kclient... 2024-09-16T01:10:42.134 INFO:tasks.kclient:Mounting kernel clients... 2024-09-16T01:10:42.134 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-16T01:10:42.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:42.134 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-09-16T01:10:42.151 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-16T01:10:42.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:42.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-16T01:10:42.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:42.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-09-16T01:10:42.333 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-09-16T01:10:42.334 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:10:42.334 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-09-16T01:10:42.334 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-09-16T01:10:42.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-16T01:10:42.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-16T01:10:42.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setuser ceph since I am not root 2024-09-16T01:10:42.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ignoring --setgroup ceph since I am not root 2024-09-16T01:10:42.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.255+0000 7fdb0d1c6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:10:42.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.316+0000 7fdb0d1c6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:42.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-16T01:10:42.386 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: ignoring --setuser ceph since I am not root 2024-09-16T01:10:42.386 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: ignoring --setgroup ceph since I am not root 2024-09-16T01:10:42.386 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-mgr[36400]: -- 172.21.15.148:0/1292439493 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb464690e0 con 0x55bb46447400 2024-09-16T01:10:42.386 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.251+0000 7f354a9fe040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:10:42.386 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.309+0000 7f354a9fe040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:42.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-16T01:10:42.459 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: ignoring --setuser ceph since I am not root 2024-09-16T01:10:42.459 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: ignoring --setgroup ceph since I am not root 2024-09-16T01:10:42.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-mgr[37511]: -- 172.21.15.193:0/273243406 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5569b9e230e0 con 0x5569b9e01400 2024-09-16T01:10:42.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.254+0000 7f395abc0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:10:42.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.310+0000 7f395abc0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:42.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.458+0000 7f395abc0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:10:42.592 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:42.713 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.474+0000 7fdb0d1c6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:10:42.713 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.546+0000 7fdb0d1c6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:10:42.714 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.712+0000 7fdb0d1c6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:42.796 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.524+0000 7f395abc0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:10:42.796 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.669+0000 7f395abc0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:42.812 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.457+0000 7f354a9fe040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:10:42.812 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.522+0000 7f354a9fe040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:10:42.812 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.666+0000 7f354a9fe040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:42.812 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:42.810+0000 7f354a9fe040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:10:43.028 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:42.857+0000 7fdb0d1c6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:10:43.101 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:42.812+0000 7f395abc0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- 172.21.15.78:0/355992321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34841036e0 msgr2=0x7f3484107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 --2- 172.21.15.78:0/355992321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34841036e0 0x7f3484107b90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3474009930 tx=0x7f347402a420 comp rx=0 tx=0).stop 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- 172.21.15.78:0/355992321 shutdown_connections 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 --2- 172.21.15.78:0/355992321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 0x7f348410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 --2- 172.21.15.78:0/355992321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34841036e0 0x7f3484107b90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 --2- 172.21.15.78:0/355992321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3484102d30 0x7f3484103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- 172.21.15.78:0/355992321 >> 172.21.15.78:0/355992321 conn(0x7f34840fc7d0 msgr2=0x7f34840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- 172.21.15.78:0/355992321 shutdown_connections 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- 172.21.15.78:0/355992321 wait complete. 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 Processor -- start 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.163+0000 7f348be0f640 1 -- start start 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34841036e0 0x7f3484100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 0x7f3484101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f34841120a0 con 0x7f3484102d30 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3484111f20 con 0x7f34841082a0 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348be0f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3484112220 con 0x7f34841036e0 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:43.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35522/0 (socket says 172.21.15.78:35522) 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 -- 172.21.15.78:0/1878040520 learned_addr learned my addr 172.21.15.78:0/1878040520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f34893aa640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34841036e0 0x7f3484100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f348a3ac640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 0x7f3484101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34841036e0 msgr2=0x7f3484100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34841036e0 0x7f3484100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 msgr2=0x7f3484101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 0x7f3484101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 -- 172.21.15.78:0/1878040520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34841a6a80 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f3489bab640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f348000d950 tx=0x7f348000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.164+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480017070 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.165+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3480009c30 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.165+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348000f6b0 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.165+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34841a6d70 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.165+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34841a7220 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.166+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3484103110 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.167+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7f34800109d0 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.167+0000 7f347affd640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 0x7f34700a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.167+0000 7f34893aa640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 msgr2=0x7f34700a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.168+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f34800cf880 con 0x7f3484102d30 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.168+0000 7f34893aa640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 0x7f34700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:10:43.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.171+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3480009de0 con 0x7f3484102d30 2024-09-16T01:10:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.369+0000 7f34893aa640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 msgr2=0x7f34700a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.369+0000 7f34893aa640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 0x7f34700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:10:43.370 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:43.187+0000 7fdb0d1c6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:10:43.370 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:43.250+0000 7fdb0d1c6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:43 smithi078 ceph-mon[30937]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:43 smithi148 ceph-mon[35030]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:43.122+0000 7f354a9fe040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:10:43.371 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:43.185+0000 7f354a9fe040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:10:43.372 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:43.368+0000 7f354a9fe040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:10:43.372 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1927597518' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-16T01:10:43.372 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:43 smithi193 ceph-mon[35701]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-16T01:10:43.372 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:43.126+0000 7f395abc0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:10:43.372 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:43.187+0000 7f395abc0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:10:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.379+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f348410a2b0 con 0x7f3484102d30 2024-09-16T01:10:43.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.380+0000 7f347affd640 1 -- 172.21.15.78:0/1878040520 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f348009bba0 con 0x7f3484102d30 2024-09-16T01:10:43.381 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 msgr2=0x7f34700a5750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 0x7f34700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 msgr2=0x7f3484100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f348000d950 tx=0x7f348000de10 comp rx=0 tx=0).stop 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 shutdown_connections 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7f34700a3290 0x7f34700a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34841082a0 0x7f3484101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34841036e0 0x7f3484100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 --2- 172.21.15.78:0/1878040520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3484102d30 0x7f3484100990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:43.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 >> 172.21.15.78:0/1878040520 conn(0x7f34840fc7d0 msgr2=0x7f3484105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:43.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 shutdown_connections 2024-09-16T01:10:43.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:43.382+0000 7f348be0f640 1 -- 172.21.15.78:0/1878040520 wait complete. 2024-09-16T01:10:43.728 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:43.443+0000 7fdb0d1c6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:10:43.796 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:43.371+0000 7f395abc0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:10:44.075 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:10:44.076 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-09-16T01:10:44.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.077 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:10:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42182sec preferred_lft 42182sec 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link noprefixroute 2024-09-16T01:10:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:10:44.135 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T01:10:44.135 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:10:44.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:44.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:44.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.276 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:10:44.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:44 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1878040520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:10:44.302 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-09-16T01:10:44.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.304 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-09-16T01:10:44.362 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-09-16T01:10:44.362 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:10:44.363 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:10:44.363 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:10:44.363 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:10:44.363 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:10:44.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:44 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1878040520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:10:44.385 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:44.251+0000 7f354a9fe040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:10:44.385 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:44.318+0000 7f354a9fe040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:10:44.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:44.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.496 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-09-16T01:10:44.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.513 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-09-16T01:10:44.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:44 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1878040520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-16T01:10:44.547 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:44.253+0000 7f395abc0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:10:44.547 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:44.319+0000 7f395abc0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:10:44.569 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:10:44.569 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:10:44.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--tmp-cephfs 2024-09-16T01:10:44.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-09-16T01:10:44.570 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:10:44.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:44.666 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:44.368+0000 7fdb0d1c6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:10:44.666 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:44.441+0000 7fdb0d1c6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:10:44.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:44.689 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-09-16T01:10:44.689 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:10:44.689 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-09-16T01:10:44.690 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:10:44.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:44.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:44.809 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:10:44.809 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-09-16T01:10:44.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-09-16T01:10:44.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T01:10:44.810 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-09-16T01:10:44.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:44.887 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:10:44.887 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /tmp/cephfs 2024-09-16T01:10:44.903 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/tmp/cephfs' 2024-09-16T01:10:44.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:10:44.903 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /tmp/cephfs 2024-09-16T01:10:44.958 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-16T01:10:44.958 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-16T01:10:44.958 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/d9b8d722-5b7a-4d90-86f4-84adec585113 2024-09-16T01:10:44.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:44.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/d9b8d722-5b7a-4d90-86f4-84adec585113 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-09-16T01:10:45.044 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-09-16T01:10:45.044 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-09-16T01:10:45.193 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-09-16T01:10:45.193 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-09-16T01:10:45.193 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.148:3300,172.21.15.193:3300" 2024-09-16T01:10:45.193 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@fa4731de-73c6-11ef-bceb-c7b262605968.cephfs=/volumes/qa/sv_1/d9b8d722-5b7a-4d90-86f4-84adec585113 2024-09-16T01:10:45.193 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.78:3300/172.21.15.148:3300/172.21.15.193:3300" will pass to kernel 2024-09-16T01:10:45.196 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:10:45.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:10:45.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /tmp/cephfs 2024-09-16T01:10:45.224 DEBUG:teuthology.orchestra.run.smithi078:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:10:45.281 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.211+0000 7fdb0d1c6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:10:45.295 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.033+0000 7f395abc0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:10:45.296 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.149+0000 7f395abc0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:10:45.297 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.032+0000 7f354a9fe040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:10:45.297 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.149+0000 7f354a9fe040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:10:45.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:10:45.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:45.322 INFO:teuthology.orchestra.run.smithi078.stdout:client24862 2024-09-16T01:10:45.323 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:10:45.323 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:10:45.381 DEBUG:teuthology.orchestra.run.smithi078:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:10:45.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:10:45.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:45.479 INFO:teuthology.orchestra.run.smithi078.stdout:client24862 2024-09-16T01:10:45.479 DEBUG:teuthology.orchestra.run.smithi078:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:10:45.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:10:45.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:45.537 INFO:teuthology.orchestra.run.smithi078.stdout:client24862 2024-09-16T01:10:45.537 DEBUG:teuthology.orchestra.run.smithi078:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/fa4731de-73c6-11ef-bceb-c7b262605968.client24862/status') 2024-09-16T01:10:45.546 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.294+0000 7f395abc0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:45.546 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.355+0000 7f395abc0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:10:45.546 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.503+0000 7f395abc0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:10:45.573 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.296+0000 7f354a9fe040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:45.573 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.357+0000 7f354a9fe040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:10:45.573 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.504+0000 7f354a9fe040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:10:45.574 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.571+0000 7f354a9fe040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:10:45.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:10:45.594 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.335+0000 7fdb0d1c6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:10:45.594 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.488+0000 7fdb0d1c6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:10:45.595 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.554+0000 7fdb0d1c6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:10:45.596 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.24862 (3)172.21.15.78:0/1366216562 2024-09-16T01:10:45.596 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2024-09-16T01:10:45.596 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2024-09-16T01:10:45.597 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2024-09-16T01:10:45.597 INFO:teuthology.orchestra.run.smithi078.stderr:71 bytes copied, 2.2488e-05 s, 3.2 MB/s 2024-09-16T01:10:45.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:10:45.604 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T01:10:45.614 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T01:10:45.614 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:45.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-16T01:10:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3641227704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b41025c0 msgr2=0x7fe8b41049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3641227704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b41025c0 0x7fe8b41049b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009900 tx=0x7fe8b002a410 comp rx=0 tx=0).stop 2024-09-16T01:10:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3641227704 shutdown_connections 2024-09-16T01:10:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3641227704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4107820 0x7fe8b4109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3641227704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b4104ef0 0x7fe8b41072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3641227704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b41025c0 0x7fe8b41049b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.714+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3641227704 >> 172.21.15.78:0/3641227704 conn(0x7fe8b40fc1e0 msgr2=0x7fe8b40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.715+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3641227704 shutdown_connections 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.715+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3641227704 wait complete. 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.715+0000 7fe8bb44c640 1 Processor -- start 2024-09-16T01:10:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.715+0000 7fe8bb44c640 1 -- start start 2024-09-16T01:10:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 0x7fe8b4195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4104ef0 0x7fe8b41961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410b640 con 0x7fe8b4107820 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410b4c0 con 0x7fe8b4104ef0 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410b7c0 con 0x7fe8b41025c0 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42008/0 (socket says 172.21.15.78:42008) 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 -- 172.21.15.78:0/3840336906 learned_addr learned my addr 172.21.15.78:0/3840336906 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b91e8640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 0x7fe8b4195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b89e7640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4104ef0 0x7fe8b41961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 msgr2=0x7fe8b4195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 0x7fe8b4195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4104ef0 msgr2=0x7fe8b41961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4104ef0 0x7fe8b41961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 -- 172.21.15.78:0/3840336906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b419d880 con 0x7fe8b4107820 2024-09-16T01:10:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b91e8640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 0x7fe8b4195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8b99e9640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00d590 tx=0x7fe8ac0114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac003970 con 0x7fe8b4107820 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b419db70 con 0x7fe8b4107820 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac004020 con 0x7fe8b4107820 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac019780 con 0x7fe8b4107820 2024-09-16T01:10:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.716+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b419e070 con 0x7fe8b4107820 2024-09-16T01:10:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.717+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b40fdcd0 con 0x7fe8b4107820 2024-09-16T01:10:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.719+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7fe8ac003b10 con 0x7fe8b4107820 2024-09-16T01:10:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.720+0000 7fe8a67fc640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 0x7fe8a00a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.720+0000 7fe8b91e8640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 msgr2=0x7fe8a00a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.720+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fe8ac0d39a0 con 0x7fe8b4107820 2024-09-16T01:10:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.720+0000 7fe8b91e8640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 0x7fe8a00a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:10:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.724+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe8ac014070 con 0x7fe8b4107820 2024-09-16T01:10:45.849 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.712+0000 7fdb0d1c6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:10:45.849 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.785+0000 7fdb0d1c6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:10:45.849 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:45.847+0000 7fdb0d1c6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:10:45.854 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.571+0000 7f395abc0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:10:45.854 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.629+0000 7f395abc0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:10:45.854 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.788+0000 7f395abc0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:10:45.854 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.852+0000 7f395abc0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:10:45.858 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.631+0000 7f354a9fe040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:10:45.859 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.791+0000 7f354a9fe040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:10:45.859 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.857+0000 7f354a9fe040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:10:45.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.921+0000 7fe8b91e8640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 msgr2=0x7fe8a00a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:45.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.921+0000 7fe8b91e8640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 0x7fe8a00a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:10:45.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.932+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fe8b402d7c0 con 0x7fe8b4107820 2024-09-16T01:10:45.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.984+0000 7fe8a67fc640 1 -- 172.21.15.78:0/3840336906 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7fe8ac09fca0 con 0x7fe8b4107820 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 msgr2=0x7fe8a00a5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 0x7fe8a00a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 msgr2=0x7fe8b419d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00d590 tx=0x7fe8ac0114a0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 shutdown_connections 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7fe8a00a3340 0x7fe8a00a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8b4107820 0x7fe8b419d1e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b4104ef0 0x7fe8b41961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 --2- 172.21.15.78:0/3840336906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b41025c0 0x7fe8b4195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:45.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 >> 172.21.15.78:0/3840336906 conn(0x7fe8b40fc1e0 msgr2=0x7fe8b4102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:45.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 shutdown_connections 2024-09-16T01:10:45.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:45.986+0000 7fe8bb44c640 1 -- 172.21.15.78:0/3840336906 wait complete. 2024-09-16T01:10:45.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-16T01:10:46.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.101+0000 7ff8af7c4640 1 -- 172.21.15.78:0/2651626242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8102620 msgr2=0x7ff8a8104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.101+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/2651626242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8102620 0x7ff8a8104a10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009900 tx=0x7ff89c02a410 comp rx=0 tx=0).stop 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- 172.21.15.78:0/2651626242 shutdown_connections 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/2651626242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a8107880 0x7ff8a8109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/2651626242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8a8104f50 0x7ff8a8107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/2651626242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8102620 0x7ff8a8104a10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- 172.21.15.78:0/2651626242 >> 172.21.15.78:0/2651626242 conn(0x7ff8a80fc1e0 msgr2=0x7ff8a80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- 172.21.15.78:0/2651626242 shutdown_connections 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- 172.21.15.78:0/2651626242 wait complete. 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 Processor -- start 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- start start 2024-09-16T01:10:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8a8102620 0x7ff8a8197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a8104f50 0x7ff8a81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 0x7ff8a819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a810d820 con 0x7ff8a8107880 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a810d6a0 con 0x7ff8a8102620 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.102+0000 7ff8af7c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a810d9a0 con 0x7ff8a8104f50 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 0x7ff8a819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 0x7ff8a819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42020/0 (socket says 172.21.15.78:42020) 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 -- 172.21.15.78:0/3421232050 learned_addr learned my addr 172.21.15.78:0/3421232050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8acd5f640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a8104f50 0x7ff8a81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8a8102620 0x7ff8a8197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a8104f50 msgr2=0x7ff8a81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a8104f50 0x7ff8a81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:46.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8a8102620 msgr2=0x7ff8a8197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8a8102620 0x7ff8a8197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a819faa0 con 0x7ff8a8107880 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8add61640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 0x7ff8a819f430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000eb40 tx=0x7ff8a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0004b20 con 0x7ff8a8107880 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a0004cc0 con 0x7ff8a8107880 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0010850 con 0x7ff8a8107880 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a819fd90 con 0x7ff8a8107880 2024-09-16T01:10:46.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.103+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a81a0280 con 0x7ff8a8107880 2024-09-16T01:10:46.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.105+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a80fdcd0 con 0x7ff8a8107880 2024-09-16T01:10:46.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.105+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7ff8a0005020 con 0x7ff8a8107880 2024-09-16T01:10:46.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.106+0000 7ff89a7fc640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 0x7ff8940a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:46.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.106+0000 7ff8ad560640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 msgr2=0x7ff8940a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:46.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.106+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7ff8a00d0480 con 0x7ff8a8107880 2024-09-16T01:10:46.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.106+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 0x7ff8940a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:10:46.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.110+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8a009cdf0 con 0x7ff8a8107880 2024-09-16T01:10:46.123 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T01:10:46.124 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:45.918+0000 7f395abc0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:10:46.124 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:46.004+0000 7f395abc0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:46.124 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:46.063+0000 7f395abc0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:10:46.124 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:46.122+0000 7f395abc0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:10:46.128 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T01:10:46.128 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:45.923+0000 7f354a9fe040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:10:46.129 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:46.006+0000 7f354a9fe040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:46.129 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:46.067+0000 7f354a9fe040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:10:46.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:45 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-16T01:10:46.185 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.014+0000 7fdb0d1c6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:10:46.185 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.083+0000 7fdb0d1c6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:10:46.185 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.182+0000 7fdb0d1c6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:10:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.307+0000 7ff8ad560640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 msgr2=0x7ff8940a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.307+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 0x7ff8940a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:10:46.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.345+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7ff8a80626d0 con 0x7ff8940a3340 2024-09-16T01:10:46.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.267+0000 7fdb0d1c6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:10:46.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.328+0000 7fdb0d1c6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:10:46.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:46.388+0000 7fdb0d1c6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:10:46.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.708+0000 7ff8ad560640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 msgr2=0x7ff8940a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1698663383 2024-09-16T01:10:46.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:46.708+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 0x7ff8940a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T01:10:47.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:47.005+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 20) ==== 138372+0+0 (secure 0 0 0) 0x7ff8a0005310 con 0x7ff8a8107880 2024-09-16T01:10:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:47.202+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 21) ==== 138040+0+0 (secure 0 0 0) 0x7ff8a0098d60 con 0x7ff8a8107880 2024-09-16T01:10:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:47.203+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 msgr2=0x7ff8940a5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:10:47.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:47.203+0000 7ff89a7fc640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/1698663383,v1:172.21.15.78:6801/1698663383] conn(0x7ff8940a3340 0x7ff8940a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:47.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.188+0000 7fdb0d1c6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:10:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T01:10:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: Standby manager daemon y restarted 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: Standby manager daemon y started 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: Standby manager daemon z restarted 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: Standby manager daemon z started 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:10:47.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:46 smithi078 ceph-mon[30937]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.296 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z[37507]: 2024-09-16T01:10:46.914+0000 7f395abc0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:10:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T01:10:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:10:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: Standby manager daemon y restarted 2024-09-16T01:10:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: Standby manager daemon y started 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: Standby manager daemon z restarted 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: Standby manager daemon z started 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:10:47.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:46 smithi193 ceph-mon[35701]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.385 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:46.127+0000 7f354a9fe040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:10:47.385 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y[36396]: 2024-09-16T01:10:46.912+0000 7f354a9fe040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3840336906' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: Standby manager daemon y restarted 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: Standby manager daemon y started 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: Standby manager daemon z restarted 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: Standby manager daemon z started 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.148:0/887102922' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-16T01:10:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-16T01:10:47.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:46 smithi148 ceph-mon[35030]: from='mgr.? 172.21.15.193:0/3200131323' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-16T01:10:47.778 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.429+0000 7fda74b18640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.429+0000 7fda74b18640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.429+0000 7fda74b18640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.429+0000 7fda74b18640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.429+0000 7fda74b18640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.432+0000 7fda7731d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.432+0000 7fda7731d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.432+0000 7fda7731d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.432+0000 7fda7731d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:47.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:47.432+0000 7fda7731d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.207+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mon.0 v2:172.21.15.78:3300/0 9 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7ff8a005e240 con 0x7ff8a8107880 2024-09-16T01:10:48.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.208+0000 7ff89a7fc640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff894035760 0x7ff894035b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.208+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7ff8a001de80 con 0x7ff894035760 2024-09-16T01:10:48.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.210+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff894035760 0x7ff894035b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.212+0000 7ff8ad560640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff894035760 0x7ff894035b40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0314e0 tx=0x7ff89c000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:48.232 INFO:teuthology.orchestra.run.smithi078.stdout:{} 2024-09-16T01:10:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.230+0000 7ff89a7fc640 1 -- 172.21.15.78:0/3421232050 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7ff8a001de80 con 0x7ff894035760 2024-09-16T01:10:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff894035760 msgr2=0x7ff894035b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff894035760 0x7ff894035b40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0314e0 tx=0x7ff89c000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 msgr2=0x7ff8a819f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 --2- 172.21.15.78:0/3421232050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8a8107880 0x7ff8a819f430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000eb40 tx=0x7ff8a000c2d0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8add61640 1 -- 172.21.15.78:0/3421232050 reap_dead start 2024-09-16T01:10:48.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 shutdown_connections 2024-09-16T01:10:48.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 >> 172.21.15.78:0/3421232050 conn(0x7ff8a80fc1e0 msgr2=0x7ff8a8103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:48.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 shutdown_connections 2024-09-16T01:10:48.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.232+0000 7ff8af7c4640 1 -- 172.21.15.78:0/3421232050 wait complete. 2024-09-16T01:10:48.245 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-16T01:10:48.259 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T01:10:48.267 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T01:10:48.268 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:48.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-09-16T01:10:48.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.218+0000 7fda71ad2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.219+0000 7fda71ad2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.219+0000 7fda71ad2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.219+0000 7fda71ad2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.278 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.219+0000 7fda71ad2640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: Active manager daemon x restarted 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: Activating manager daemon x 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: mgrmap e21: x(active, starting, since 0.01284s), standbys: y, z 2024-09-16T01:10:48.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:10:48.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:10:48.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: Manager daemon x is now available 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:10:48.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: Active manager daemon x restarted 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: Activating manager daemon x 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: mgrmap e21: x(active, starting, since 0.01284s), standbys: y, z 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:10:48.286 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:10:48.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:10:48.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: Manager daemon x is now available 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:10:48.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:48 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: Active manager daemon x restarted 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: Activating manager daemon x 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: osdmap e94: 12 total, 12 up, 12 in 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: mgrmap e21: x(active, starting, since 0.01284s), standbys: y, z 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-16T01:10:48.313 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:10:48.314 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-16T01:10:48.315 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-16T01:10:48.316 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: Manager daemon x is now available 2024-09-16T01:10:48.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:48.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-16T01:10:48.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:48 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-16T01:10:48.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 -- 172.21.15.78:0/3286940035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d4101690 msgr2=0x7ff8d4101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3286940035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d4101690 0x7ff8d4101b40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009ca0 tx=0x7ff8cc02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 -- 172.21.15.78:0/3286940035 shutdown_connections 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3286940035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d4101690 0x7ff8d4101b40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3286940035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4100cf0 0x7ff8d4101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3286940035 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d41003b0 0x7ff8d41007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 -- 172.21.15.78:0/3286940035 >> 172.21.15.78:0/3286940035 conn(0x7ff8d40fc1e0 msgr2=0x7ff8d40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 -- 172.21.15.78:0/3286940035 shutdown_connections 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.368+0000 7ff8da91f640 1 -- 172.21.15.78:0/3286940035 wait complete. 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 Processor -- start 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 -- start start 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d4100cf0 0x7ff8d419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4101690 0x7ff8d41a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d410f9b0 con 0x7ff8d41003b0 2024-09-16T01:10:48.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d410f830 con 0x7ff8d4100cf0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8da91f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d410fb30 con 0x7ff8d4101690 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42176/0 (socket says 172.21.15.78:42176) 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8d3fff640 1 -- 172.21.15.78:0/3968225438 learned_addr learned my addr 172.21.15.78:0/3968225438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8d8ebc640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4101690 0x7ff8d41a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.369+0000 7ff8d37fe640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d4100cf0 0x7ff8d419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 -- 172.21.15.78:0/3968225438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4101690 msgr2=0x7ff8d41a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4101690 0x7ff8d41a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 -- 172.21.15.78:0/3968225438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d4100cf0 msgr2=0x7ff8d419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d4100cf0 0x7ff8d419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 -- 172.21.15.78:0/3968225438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d41a1b70 con 0x7ff8d41003b0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d3fff640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400d950 tx=0x7ff8c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:48.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4017030 con 0x7ff8d41003b0 2024-09-16T01:10:48.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c4004020 con 0x7ff8d41003b0 2024-09-16T01:10:48.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40047a0 con 0x7ff8d41003b0 2024-09-16T01:10:48.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d41a1e60 con 0x7ff8d41003b0 2024-09-16T01:10:48.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.370+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d41a22d0 con 0x7ff8d41003b0 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.371+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d410b100 con 0x7ff8d41003b0 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.372+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7ff8c40041d0 con 0x7ff8d41003b0 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.373+0000 7ff8d17fa640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 0x7ff8bc0a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.373+0000 7ff8d37fe640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 0x7ff8bc0a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.373+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff8c40cf960 con 0x7ff8d41003b0 2024-09-16T01:10:48.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.373+0000 7ff8d37fe640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 0x7ff8bc0a5640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff8d419b460 tx=0x7ff8c8005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:48.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.376+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8c409c540 con 0x7ff8d41003b0 2024-09-16T01:10:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.561+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7ff8d402d7c0 con 0x7ff8d41003b0 2024-09-16T01:10:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.565+0000 7ff8d17fa640 1 -- 172.21.15.78:0/3968225438 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v31)=0 v31) ==== 129+0+0 (secure 0 0 0) 0x7ff8c409be40 con 0x7ff8d41003b0 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 msgr2=0x7ff8bc0a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 0x7ff8bc0a5640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff8d419b460 tx=0x7ff8c8005f90 comp rx=0 tx=0).stop 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 msgr2=0x7ff8d4199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400d950 tx=0x7ff8c400de10 comp rx=0 tx=0).stop 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 shutdown_connections 2024-09-16T01:10:48.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff8bc0a3180 0x7ff8bc0a5640 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8d4101690 0x7ff8d41a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d4100cf0 0x7ff8d419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 --2- 172.21.15.78:0/3968225438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d41003b0 0x7ff8d4199f50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 >> 172.21.15.78:0/3968225438 conn(0x7ff8d40fc1e0 msgr2=0x7ff8d40fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 shutdown_connections 2024-09-16T01:10:48.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.567+0000 7ff8da91f640 1 -- 172.21.15.78:0/3968225438 wait complete. 2024-09-16T01:10:48.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-09-16T01:10:48.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 -- 172.21.15.78:0/1227063329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc1025d0 msgr2=0x7f89fc1049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/1227063329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc1025d0 0x7f89fc1049c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009900 tx=0x7f89ec02a410 comp rx=0 tx=0).stop 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 -- 172.21.15.78:0/1227063329 shutdown_connections 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/1227063329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc107830 0x7f89fc109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/1227063329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc104f00 0x7f89fc1072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/1227063329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc1025d0 0x7f89fc1049c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 -- 172.21.15.78:0/1227063329 >> 172.21.15.78:0/1227063329 conn(0x7f89fc0fc1b0 msgr2=0x7f89fc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:48.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 -- 172.21.15.78:0/1227063329 shutdown_connections 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.722+0000 7f8a00cb7640 1 -- 172.21.15.78:0/1227063329 wait complete. 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 Processor -- start 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 -- start start 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 0x7f89fc197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 0x7f89fc198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc10d810 con 0x7f89fc107830 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc10d690 con 0x7f89fc1025d0 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f8a00cb7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc10d990 con 0x7f89fc104f00 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42194/0 (socket says 172.21.15.78:42194) 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 -- 172.21.15.78:0/3283766918 learned_addr learned my addr 172.21.15.78:0/3283766918 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fa59c640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 0x7f89fc197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89f1d9b640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 0x7f89fc198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 -- 172.21.15.78:0/3283766918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 msgr2=0x7f89fc198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 0x7f89fc198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 -- 172.21.15.78:0/3283766918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 msgr2=0x7f89fc197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 0x7f89fc197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fad9d640 1 -- 172.21.15.78:0/3283766918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc19fa10 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89fa59c640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 0x7f89fc197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.723+0000 7f89f1d9b640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 0x7f89fc198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f89fad9d640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f89f400e9d0 tx=0x7f89f400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f400cd80 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f4005020 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc19fd00 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f40116d0 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.724+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89fc1a01a0 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.725+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89fc0fdca0 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.726+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7f89f40051d0 con 0x7f89fc107830 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.727+0000 7f89f3fff640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 0x7f89e80a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:48.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.727+0000 7f89fa59c640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 0x7f89e80a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:48.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.727+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f89f40cff40 con 0x7f89fc107830 2024-09-16T01:10:48.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.727+0000 7f89fa59c640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 0x7f89e80a55c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0314e0 tx=0x7f89ec000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:48.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.730+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f89f4016030 con 0x7f89fc107830 2024-09-16T01:10:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.908+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f89fc02d7c0 con 0x7f89fc107830 2024-09-16T01:10:48.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.912+0000 7f89f3fff640 1 -- 172.21.15.78:0/3283766918 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7f89f409caa0 con 0x7f89fc107830 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 msgr2=0x7f89e80a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 0x7f89e80a55c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0314e0 tx=0x7f89ec000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 msgr2=0x7f89fc19f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f89f400e9d0 tx=0x7f89f400ee90 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 shutdown_connections 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89e80a3100 0x7f89e80a55c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89fc107830 0x7f89fc19f3a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89fc104f00 0x7f89fc198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 --2- 172.21.15.78:0/3283766918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89fc1025d0 0x7f89fc197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.915+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 >> 172.21.15.78:0/3283766918 conn(0x7f89fc0fc1b0 msgr2=0x7f89fc102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.916+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 shutdown_connections 2024-09-16T01:10:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:48.916+0000 7f8a00cb7640 1 -- 172.21.15.78:0/3283766918 wait complete. 2024-09-16T01:10:48.930 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.557+0000 7fda569dd640 -1 log_channel(cephadm) log [ERR] : [16/Sep/2024:01:10:48] ENGINE Error in HTTPServer.serve 2024-09-16T01:10:48.930 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: Traceback (most recent call last): 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._connections.run(self.expiration_interval) 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._run(expiration_interval) 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: s = self.context.wrap_socket( 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:10:48.931 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: return self.sslsocket_class._create( 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self.do_handshake() 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: self._sslobj.do_handshake() 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:10:48.932 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:10:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:10:48.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-09-16T01:10:49.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 -- 172.21.15.78:0/3938948553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 msgr2=0x7ff7681020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3938948553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff7681020b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff750009900 tx=0x7ff75002a410 comp rx=0 tx=0).stop 2024-09-16T01:10:49.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 -- 172.21.15.78:0/3938948553 shutdown_connections 2024-09-16T01:10:49.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3938948553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 0x7ff768108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3938948553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff7681020b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3938948553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff768100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 -- 172.21.15.78:0/3938948553 >> 172.21.15.78:0/3938948553 conn(0x7ff7680fc1e0 msgr2=0x7ff7680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:49.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 -- 172.21.15.78:0/3938948553 shutdown_connections 2024-09-16T01:10:49.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.039+0000 7ff76cd78640 1 -- 172.21.15.78:0/3938948553 wait complete. 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 Processor -- start 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 -- start start 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff76806a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 0x7ff76819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff76810c710 con 0x7ff768101c30 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff76810c590 con 0x7ff7681025f0 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76cd78640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff76810c890 con 0x7ff768100a40 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42212/0 (socket says 172.21.15.78:42212) 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 -- 172.21.15.78:0/3496819614 learned_addr learned my addr 172.21.15.78:0/3496819614 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76659c640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff76806a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff766d9d640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 0x7ff76819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 -- 172.21.15.78:0/3496819614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 msgr2=0x7ff76806a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff76806a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 -- 172.21.15.78:0/3496819614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 msgr2=0x7ff76819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 0x7ff76819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff765d9b640 1 -- 172.21.15.78:0/3496819614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76819fa50 con 0x7ff768101c30 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.040+0000 7ff76659c640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff76806a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff765d9b640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff750031440 tx=0x7ff750032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75003f030 con 0x7ff768101c30 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff750003ce0 con 0x7ff768101c30 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7681968f0 con 0x7ff768101c30 2024-09-16T01:10:49.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750035070 con 0x7ff768101c30 2024-09-16T01:10:49.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.041+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff768196e30 con 0x7ff768101c30 2024-09-16T01:10:49.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.042+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff768107d60 con 0x7ff768101c30 2024-09-16T01:10:49.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.043+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7ff750042030 con 0x7ff768101c30 2024-09-16T01:10:49.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.044+0000 7ff75f7fe640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 0x7ff7540a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.044+0000 7ff76659c640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 0x7ff7540a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.044+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff7500efe80 con 0x7ff768101c30 2024-09-16T01:10:49.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.044+0000 7ff76659c640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 0x7ff7540a5640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff768101a90 tx=0x7ff758005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.048+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff7500bc390 con 0x7ff768101c30 2024-09-16T01:10:49.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.270+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7ff76802d7c0 con 0x7ff768101c30 2024-09-16T01:10:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.274+0000 7ff75f7fe640 1 -- 172.21.15.78:0/3496819614 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7ff7500c0690 con 0x7ff768101c30 2024-09-16T01:10:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 msgr2=0x7ff7540a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 0x7ff7540a5640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff768101a90 tx=0x7ff758005f90 comp rx=0 tx=0).stop 2024-09-16T01:10:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 msgr2=0x7ff76819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff750031440 tx=0x7ff750032ce0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 shutdown_connections 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff7540a3180 0x7ff7540a5640 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7681025f0 0x7ff76819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff768101c30 0x7ff76819cb20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 --2- 172.21.15.78:0/3496819614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff768100a40 0x7ff76806a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 >> 172.21.15.78:0/3496819614 conn(0x7ff7680fc1e0 msgr2=0x7ff768104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 shutdown_connections 2024-09-16T01:10:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.277+0000 7ff76cd78640 1 -- 172.21.15.78:0/3496819614 wait complete. 2024-09-16T01:10:49.294 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-16T01:10:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.399+0000 7f2db67a6640 1 -- 172.21.15.78:0/3694287341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0101c30 msgr2=0x7f2db01020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.399+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3694287341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0101c30 0x7f2db01020b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009900 tx=0x7f2da402a410 comp rx=0 tx=0).stop 2024-09-16T01:10:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- 172.21.15.78:0/3694287341 shutdown_connections 2024-09-16T01:10:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3694287341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 0x7f2db0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3694287341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0101c30 0x7f2db01020b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3694287341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0100a40 0x7f2db0100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- 172.21.15.78:0/3694287341 >> 172.21.15.78:0/3694287341 conn(0x7f2db00fc1e0 msgr2=0x7f2db00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- 172.21.15.78:0/3694287341 shutdown_connections 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- 172.21.15.78:0/3694287341 wait complete. 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 Processor -- start 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- start start 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0101c30 0x7f2db0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 0x7f2db019f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2db010c720 con 0x7f2db0100a40 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2db010c5a0 con 0x7f2db0101c30 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.400+0000 7f2db67a6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2db010c8a0 con 0x7f2db01025f0 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42264/0 (socket says 172.21.15.78:42264) 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 -- 172.21.15.78:0/3491569316 learned_addr learned my addr 172.21.15.78:0/3491569316 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daf7fe640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0101c30 0x7f2db0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2db4d43640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 0x7f2db019f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 -- 172.21.15.78:0/3491569316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 msgr2=0x7f2db019f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 0x7f2db019f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 -- 172.21.15.78:0/3491569316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0101c30 msgr2=0x7f2db0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0101c30 0x7f2db0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 -- 172.21.15.78:0/3491569316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db019fa30 con 0x7f2db0100a40 2024-09-16T01:10:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2daffff640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9c0 tx=0x7f2da000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da000cd80 con 0x7f2db0100a40 2024-09-16T01:10:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da00035f0 con 0x7f2db0100a40 2024-09-16T01:10:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db019fd20 con 0x7f2db0100a40 2024-09-16T01:10:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0010690 con 0x7f2db0100a40 2024-09-16T01:10:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.401+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db01a0210 con 0x7f2db0100a40 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.402+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db01a04a0 con 0x7f2db0100a40 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.403+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7f2da00037a0 con 0x7f2db0100a40 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.404+0000 7f2dad7fa640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 0x7f2d980a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.404+0000 7f2daf7fe640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 0x7f2d980a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.404+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2da00cfe30 con 0x7f2db0100a40 2024-09-16T01:10:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.405+0000 7f2daf7fe640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 0x7f2d980a55c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2da40314e0 tx=0x7f2da4000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.407+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2da009ca20 con 0x7f2db0100a40 2024-09-16T01:10:49.421 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: mgrmap e22: x(active, since 1.01869s), standbys: y, z 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: [16/Sep/2024:01:10:48] ENGINE Bus STARTING 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: [16/Sep/2024:01:10:48] ENGINE Serving on http://172.21.15.78:8765 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: [16/Sep/2024:01:10:48] ENGINE Serving on https://172.21.15.78:7150 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: [16/Sep/2024:01:10:48] ENGINE Bus STARTED 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: [16/Sep/2024:01:10:48] ENGINE Error in HTTPServer.serve 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: Traceback (most recent call last): 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._connections.run(self.expiration_interval) 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._run(expiration_interval) 2024-09-16T01:10:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: s = self.context.wrap_socket( 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: return self.sslsocket_class._create( 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: self.do_handshake() 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._sslobj.do_handshake() 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3968225438' entity='client.admin' 2024-09-16T01:10:49.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3283766918' entity='client.admin' 2024-09-16T01:10:49.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:49 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: mgrmap e22: x(active, since 1.01869s), standbys: y, z 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: [16/Sep/2024:01:10:48] ENGINE Bus STARTING 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: [16/Sep/2024:01:10:48] ENGINE Serving on http://172.21.15.78:8765 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: [16/Sep/2024:01:10:48] ENGINE Serving on https://172.21.15.78:7150 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: [16/Sep/2024:01:10:48] ENGINE Bus STARTED 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: [16/Sep/2024:01:10:48] ENGINE Error in HTTPServer.serve 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: Traceback (most recent call last): 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._connections.run(self.expiration_interval) 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._run(expiration_interval) 2024-09-16T01:10:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: s = self.context.wrap_socket( 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: return self.sslsocket_class._create( 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: self.do_handshake() 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: self._sslobj.do_handshake() 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:10:49.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3968225438' entity='client.admin' 2024-09-16T01:10:49.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3283766918' entity='client.admin' 2024-09-16T01:10:49.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:49 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.603+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2db00fe3d0 con 0x7f2d980a3100 2024-09-16T01:10:49.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: mgrmap e22: x(active, since 1.01869s), standbys: y, z 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: [16/Sep/2024:01:10:48] ENGINE Bus STARTING 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: [16/Sep/2024:01:10:48] ENGINE Serving on http://172.21.15.78:8765 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: [16/Sep/2024:01:10:48] ENGINE Serving on https://172.21.15.78:7150 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: [16/Sep/2024:01:10:48] ENGINE Bus STARTED 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: [16/Sep/2024:01:10:48] ENGINE Error in HTTPServer.serve 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout: Traceback (most recent call last): 2024-09-16T01:10:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-16T01:10:49.637 INFO:journalctl@ceph.mon.b.smithi148.stdout: self._connections.run(self.expiration_interval) 2024-09-16T01:10:49.637 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-16T01:10:49.637 INFO:journalctl@ceph.mon.b.smithi148.stdout: self._run(expiration_interval) 2024-09-16T01:10:49.637 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-16T01:10:49.637 INFO:journalctl@ceph.mon.b.smithi148.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: s = self.context.wrap_socket( 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: return self.sslsocket_class._create( 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: self.do_handshake() 2024-09-16T01:10:49.638 INFO:journalctl@ceph.mon.b.smithi148.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout: self._sslobj.do_handshake() 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3968225438' entity='client.admin' 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3283766918' entity='client.admin' 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:49 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.691+0000 7f2dad7fa640 1 -- 172.21.15.78:0/3491569316 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2db00fe3d0 con 0x7f2d980a3100 2024-09-16T01:10:49.694 INFO:teuthology.orchestra.run.smithi078.stdout:Schedule set for path / 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 msgr2=0x7f2d980a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 0x7f2d980a55c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2da40314e0 tx=0x7f2da4000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 msgr2=0x7f2db0197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9c0 tx=0x7f2da000ee80 comp rx=0 tx=0).stop 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 shutdown_connections 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2d980a3100 0x7f2d980a55c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2db01025f0 0x7f2db019f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0101c30 0x7f2db0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 --2- 172.21.15.78:0/3491569316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2db0100a40 0x7f2db0197e10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 >> 172.21.15.78:0/3491569316 conn(0x7f2db00fc1e0 msgr2=0x7f2db00fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 shutdown_connections 2024-09-16T01:10:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.694+0000 7f2db67a6640 1 -- 172.21.15.78:0/3491569316 wait complete. 2024-09-16T01:10:49.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- 172.21.15.78:0/2910310942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 msgr2=0x7ffad0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- 172.21.15.78:0/2910310942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad0109cb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009ca0 tx=0x7ffac802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- 172.21.15.78:0/2910310942 shutdown_connections 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- 172.21.15.78:0/2910310942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad0109cb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- 172.21.15.78:0/2910310942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- 172.21.15.78:0/2910310942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 0x7ffad0104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- 172.21.15.78:0/2910310942 >> 172.21.15.78:0/2910310942 conn(0x7ffad00fc1e0 msgr2=0x7ffad00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- 172.21.15.78:0/2910310942 shutdown_connections 2024-09-16T01:10:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- 172.21.15.78:0/2910310942 wait complete. 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 Processor -- start 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 -- start start 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 0x7ffad0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.812+0000 7ffad740e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad740e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010d830 con 0x7ffad01078c0 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad740e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010d6b0 con 0x7ffad0104f90 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad740e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010d9b0 con 0x7ffad0102660 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42280/0 (socket says 172.21.15.78:42280) 2024-09-16T01:10:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 -- 172.21.15.78:0/1992769272 learned_addr learned my addr 172.21.15.78:0/1992769272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad49a9640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad51aa640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 0x7ffad0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 -- 172.21.15.78:0/1992769272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 msgr2=0x7ffad0197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 0x7ffad0197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 -- 172.21.15.78:0/1992769272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 msgr2=0x7ffad0198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 -- 172.21.15.78:0/1992769272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad019fa00 con 0x7ffad01078c0 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad49a9640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:10:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.813+0000 7ffad59ab640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffac8031ae0 tx=0x7ffac80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.815+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8003bf0 con 0x7ffad01078c0 2024-09-16T01:10:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.815+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac8003d90 con 0x7ffad01078c0 2024-09-16T01:10:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.815+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac802fce0 con 0x7ffad01078c0 2024-09-16T01:10:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.815+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad019fc90 con 0x7ffad01078c0 2024-09-16T01:10:49.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.816+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad01a0190 con 0x7ffad01078c0 2024-09-16T01:10:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.816+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad00fdcd0 con 0x7ffad01078c0 2024-09-16T01:10:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.818+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) ==== 138167+0+0 (secure 0 0 0) 0x7ffac8004b40 con 0x7ffad01078c0 2024-09-16T01:10:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.818+0000 7ffac67fc640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 0x7ffabc0a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.818+0000 7ffad51aa640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 0x7ffabc0a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.818+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffac80f0fc0 con 0x7ffad01078c0 2024-09-16T01:10:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.819+0000 7ffad51aa640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 0x7ffabc0a5640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffacc005d00 tx=0x7ffacc00b410 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:49.826+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffac80bdb20 con 0x7ffad01078c0 2024-09-16T01:10:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.033+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ffad00626d0 con 0x7ffabc0a3180 2024-09-16T01:10:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.090+0000 7ffac67fc640 1 -- 172.21.15.78:0/1992769272 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ffad00626d0 con 0x7ffabc0a3180 2024-09-16T01:10:50.093 INFO:teuthology.orchestra.run.smithi078.stdout:Retention added to path / 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 msgr2=0x7ffabc0a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 0x7ffabc0a5640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffacc005d00 tx=0x7ffacc00b410 comp rx=0 tx=0).stop 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 msgr2=0x7ffad019f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffac8031ae0 tx=0x7ffac80319f0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 shutdown_connections 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffabc0a3180 0x7ffabc0a5640 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad01078c0 0x7ffad019f390 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffad0104f90 0x7ffad0198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 --2- 172.21.15.78:0/1992769272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffad0102660 0x7ffad0197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 >> 172.21.15.78:0/1992769272 conn(0x7ffad00fc1e0 msgr2=0x7ffad0102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 shutdown_connections 2024-09-16T01:10:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.093+0000 7ffad740e640 1 -- 172.21.15.78:0/1992769272 wait complete. 2024-09-16T01:10:50.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-16T01:10:50.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 -- 172.21.15.78:0/3769491753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8710107830 msgr2=0x7f8710109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 --2- 172.21.15.78:0/3769491753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8710107830 0x7f8710109ca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8708009ca0 tx=0x7f870802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 -- 172.21.15.78:0/3769491753 shutdown_connections 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 --2- 172.21.15.78:0/3769491753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8710107830 0x7f8710109ca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 --2- 172.21.15.78:0/3769491753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710104f00 0x7f87101072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 --2- 172.21.15.78:0/3769491753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87101025d0 0x7f87101049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 -- 172.21.15.78:0/3769491753 >> 172.21.15.78:0/3769491753 conn(0x7f87100fc1b0 msgr2=0x7f87100fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 -- 172.21.15.78:0/3769491753 shutdown_connections 2024-09-16T01:10:50.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.215+0000 7f87155c4640 1 -- 172.21.15.78:0/3769491753 wait complete. 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 Processor -- start 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 -- start start 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8710104f00 0x7f87101983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710107830 0x7f871019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f871010d810 con 0x7f87101025d0 2024-09-16T01:10:50.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f871010d690 con 0x7f8710107830 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f87155c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f871010d990 con 0x7f8710104f00 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42312/0 (socket says 172.21.15.78:42312) 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 -- 172.21.15.78:0/55566069 learned_addr learned my addr 172.21.15.78:0/55566069 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870e7fc640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8710104f00 0x7f87101983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870f7fe640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710107830 0x7f871019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 -- 172.21.15.78:0/55566069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8710104f00 msgr2=0x7f87101983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8710104f00 0x7f87101983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 -- 172.21.15.78:0/55566069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710107830 msgr2=0x7f871019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710107830 0x7f871019f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.216+0000 7f870effd640 1 -- 172.21.15.78:0/55566069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871019fa80 con 0x7f87101025d0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f870effd640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9f0 tx=0x7f870000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870000cd80 con 0x7f87101025d0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871019fd70 con 0x7f87101025d0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8700005500 con 0x7f87101025d0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700010690 con 0x7f87101025d0 2024-09-16T01:10:50.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.217+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101a0200 con 0x7f87101025d0 2024-09-16T01:10:50.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.218+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87100fdca0 con 0x7f87101025d0 2024-09-16T01:10:50.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.220+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 23) ==== 138526+0+0 (secure 0 0 0) 0x7f87000046c0 con 0x7f87101025d0 2024-09-16T01:10:50.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.220+0000 7f86f7fff640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 0x7f86f80a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.221+0000 7f870e7fc640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 0x7f86f80a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.221+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f87000d0610 con 0x7f87101025d0 2024-09-16T01:10:50.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.221+0000 7f870e7fc640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 0x7f86f80a59c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8710199370 tx=0x7f8704005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:50.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.228+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f870009caf0 con 0x7f87101025d0 2024-09-16T01:10:50.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.427+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8710062630 con 0x7f86f80a3500 2024-09-16T01:10:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.433+0000 7f86f7fff640 1 -- 172.21.15.78:0/55566069 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f8710062630 con 0x7f86f80a3500 2024-09-16T01:10:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-16T00:00:00", "created": "2024-09-16T01:10:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 msgr2=0x7f86f80a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 0x7f86f80a59c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8710199370 tx=0x7f8704005f90 comp rx=0 tx=0).stop 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 msgr2=0x7f8710197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9f0 tx=0x7f870000edf0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 shutdown_connections 2024-09-16T01:10:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f86f80a3500 0x7f86f80a59c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8710107830 0x7f871019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8710104f00 0x7f87101983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 --2- 172.21.15.78:0/55566069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87101025d0 0x7f8710197e60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 >> 172.21.15.78:0/55566069 conn(0x7f87100fc1b0 msgr2=0x7f8710102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 shutdown_connections 2024-09-16T01:10:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.437+0000 7f87155c4640 1 -- 172.21.15.78:0/55566069 wait complete. 2024-09-16T01:10:50.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:50.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3496819614' entity='client.admin' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.452 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:50 smithi078 ceph-mon[30937]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-09-16T01:10:50.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3496819614' entity='client.admin' 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:50.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:10:50.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:10:50.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:10:50.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:50 smithi193 ceph-mon[35701]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1455048666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e80ffe20 msgr2=0x7fe5e8100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1455048666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8100220 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009900 tx=0x7fe5d802a410 comp rx=0 tx=0).stop 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1455048666 shutdown_connections 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1455048666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e81037b0 0x7fe5e810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1455048666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e8102df0 0x7fe5e8103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1455048666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8100220 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1455048666 >> 172.21.15.78:0/1455048666 conn(0x7fe5e80fb980 msgr2=0x7fe5e80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1455048666 shutdown_connections 2024-09-16T01:10:50.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.580+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1455048666 wait complete. 2024-09-16T01:10:50.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.581+0000 7fe5ed01e640 1 Processor -- start 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.581+0000 7fe5ed01e640 1 -- start start 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.581+0000 7fe5ed01e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.581+0000 7fe5ed01e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.581+0000 7fe5ed01e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e81037b0 0x7fe5e819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5ed01e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810d7d0 con 0x7fe5e8102df0 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5ed01e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810d650 con 0x7fe5e80ffe20 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5ed01e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810d950 con 0x7fe5e81037b0 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42338/0 (socket says 172.21.15.78:42338) 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 -- 172.21.15.78:0/1082135868 learned_addr learned my addr 172.21.15.78:0/1082135868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e759e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e81037b0 0x7fe5e819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e6d9d640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 -- 172.21.15.78:0/1082135868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e81037b0 msgr2=0x7fe5e819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e81037b0 0x7fe5e819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 -- 172.21.15.78:0/1082135868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e80ffe20 msgr2=0x7fe5e8197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 -- 172.21.15.78:0/1082135868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e819f9c0 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5e659c640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c920 tx=0x7fe5dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.582+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc007c70 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.583+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e819fcb0 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.583+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5dc007e10 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.583+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc018890 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.583+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e81a01d0 con 0x7fe5e8102df0 2024-09-16T01:10:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.584+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e8108f70 con 0x7fe5e8102df0 2024-09-16T01:10:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.585+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 23) ==== 138526+0+0 (secure 0 0 0) 0x7fe5dc018a70 con 0x7fe5e8102df0 2024-09-16T01:10:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.585+0000 7fe5cffff640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 0x7fe5d00a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.586+0000 7fe5e6d9d640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 0x7fe5d00a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.586+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe5dc0d0370 con 0x7fe5e8102df0 2024-09-16T01:10:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.586+0000 7fe5e6d9d640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 0x7fe5d00a59c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8005ab0 tx=0x7fe5d8000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.590+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe5dc014030 con 0x7fe5e8102df0 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3496819614' entity='client.admin' 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:10:50.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Updating smithi078:/etc/ceph/ceph.conf 2024-09-16T01:10:50.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-16T01:10:50.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: Updating smithi193:/etc/ceph/ceph.conf 2024-09-16T01:10:50.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:50.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:50 smithi148 ceph-mon[35030]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-09-16T01:10:50.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.794+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe5e8062630 con 0x7fe5d00a3500 2024-09-16T01:10:50.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.801+0000 7fe5cffff640 1 -- 172.21.15.78:0/1082135868 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe5e8062630 con 0x7fe5d00a3500 2024-09-16T01:10:50.803 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1m 6m3h 2024-09-16T01:10:50.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 msgr2=0x7fe5d00a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 0x7fe5d00a59c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8005ab0 tx=0x7fe5d8000bc0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 msgr2=0x7fe5e81982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:50.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c920 tx=0x7fe5dc00cde0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 shutdown_connections 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe5d00a3500 0x7fe5d00a59c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5e81037b0 0x7fe5e819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5e8102df0 0x7fe5e81982f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 --2- 172.21.15.78:0/1082135868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5e80ffe20 0x7fe5e8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 >> 172.21.15.78:0/1082135868 conn(0x7fe5e80fb980 msgr2=0x7fe5e8106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 shutdown_connections 2024-09-16T01:10:50.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:50.803+0000 7fe5ed01e640 1 -- 172.21.15.78:0/1082135868 wait complete. 2024-09-16T01:10:50.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-16T01:10:50.847 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-16T01:10:50.858 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-16T01:10:50.868 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:10:50.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-16T01:10:50.891 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-16T01:10:50.905 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-16T01:10:50.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:10:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 -- 172.21.15.78:0/2918947927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c101420 msgr2=0x7f845c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 --2- 172.21.15.78:0/2918947927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c101420 0x7f845c10d950 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8454009cd0 tx=0x7f845402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 -- 172.21.15.78:0/2918947927 shutdown_connections 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 --2- 172.21.15.78:0/2918947927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c101420 0x7f845c10d950 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 --2- 172.21.15.78:0/2918947927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c100a80 0x7f845c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 --2- 172.21.15.78:0/2918947927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c106a80 0x7f845c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 -- 172.21.15.78:0/2918947927 >> 172.21.15.78:0/2918947927 conn(0x7f845c0fc7a0 msgr2=0x7f845c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 -- 172.21.15.78:0/2918947927 shutdown_connections 2024-09-16T01:10:51.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.675+0000 7f846109a640 1 -- 172.21.15.78:0/2918947927 wait complete. 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 Processor -- start 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 -- start start 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c101420 0x7f845c102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c106a80 0x7f845c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f845c110e80 con 0x7f845c100a80 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f845c110d00 con 0x7f845c101420 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f846109a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f845c111000 con 0x7f845c106a80 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.676+0000 7f845ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42354/0 (socket says 172.21.15.78:42354) 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 -- 172.21.15.78:0/1207147969 learned_addr learned my addr 172.21.15.78:0/1207147969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:51.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845259c640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c101420 0x7f845c102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845b59e640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c106a80 0x7f845c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 -- 172.21.15.78:0/1207147969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c106a80 msgr2=0x7f845c1030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c106a80 0x7f845c1030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 -- 172.21.15.78:0/1207147969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c101420 msgr2=0x7f845c102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c101420 0x7f845c102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 -- 172.21.15.78:0/1207147969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845c1a68f0 con 0x7f845c100a80 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f845ad9d640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f844c00e9f0 tx=0x7f844c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c00cd80 con 0x7f845c100a80 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845c1a6be0 con 0x7f845c100a80 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f844c010910 con 0x7f845c100a80 2024-09-16T01:10:51.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c0188f0 con 0x7f845c100a80 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.677+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845c1a7170 con 0x7f845c100a80 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.678+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845c100ee0 con 0x7f845c100a80 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.679+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 23) ==== 138526+0+0 (secure 0 0 0) 0x7f844c010ac0 con 0x7f845c100a80 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.680+0000 7f8458d99640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 0x7f84480a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.680+0000 7f845259c640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 0x7f84480a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.680+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f844c0d05d0 con 0x7f845c100a80 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.680+0000 7f845259c640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 0x7f84480a5900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f845c1040e0 tx=0x7f844400b390 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:51.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.684+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f844c09cab0 con 0x7f845c100a80 2024-09-16T01:10:51.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.850+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f845c06a790 con 0x7f84480a3440 2024-09-16T01:10:51.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.860+0000 7f8458d99640 1 -- 172.21.15.78:0/1207147969 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339711 (secure 0 0 0) 0x7f845c06a790 con 0x7f84480a3440 2024-09-16T01:10:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:51.868 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 msgr2=0x7f84480a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 0x7f84480a5900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f845c1040e0 tx=0x7f844400b390 comp rx=0 tx=0).stop 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 msgr2=0x7f845c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f844c00e9f0 tx=0x7f844c00eeb0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 shutdown_connections 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f84480a3440 0x7f84480a5900 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f845c106a80 0x7f845c1030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845c101420 0x7f845c102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 --2- 172.21.15.78:0/1207147969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845c100a80 0x7f845c102650 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 >> 172.21.15.78:0/1207147969 conn(0x7f845c0fc7a0 msgr2=0x7f845c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.863+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 shutdown_connections 2024-09-16T01:10:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:51.864+0000 7f846109a640 1 -- 172.21.15.78:0/1207147969 wait complete. 2024-09-16T01:10:52.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:52.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:52.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: Updating smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:10:52.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:10:52.621 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-16T01:10:51.210047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1189448,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":872,"num_read_kb":592,"num_write":481,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378672,"kb_used_data":12744,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563152,"statfs":{"total":1151940427776,"available":1151552667648,"internally_reserved":0,"allocated":13049856,"data_stored":9467190,"data_compressed":547200,"data_compressed_allocated":741376,"data_compressed_original":3883008,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991609"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287576+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:10:47.287576+0000","last_peered":"2024-09-16T01:10:47.287576+0000","last_clean":"2024-09-16T01:10:47.287576+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:10:47.287576+0000","last_undegraded":"2024-09-16T01:10:47.287576+0000","last_fullsized":"2024-09-16T01:10:47.287576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198446+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:10:47.198446+0000","last_peered":"2024-09-16T01:10:47.198446+0000","last_clean":"2024-09-16T01:10:47.198446+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:10:47.198446+0000","last_undegraded":"2024-09-16T01:10:47.198446+0000","last_fullsized":"2024-09-16T01:10:47.198446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201248+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:10:47.201248+0000","last_peered":"2024-09-16T01:10:47.201248+0000","last_clean":"2024-09-16T01:10:47.201248+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:10:47.201248+0000","last_undegraded":"2024-09-16T01:10:47.201248+0000","last_fullsized":"2024-09-16T01:10:47.201248+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198687+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:10:47.198687+0000","last_peered":"2024-09-16T01:10:47.198687+0000","last_clean":"2024-09-16T01:10:47.198687+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:10:47.198687+0000","last_undegraded":"2024-09-16T01:10:47.198687+0000","last_fullsized":"2024-09-16T01:10:47.198687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526283+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:10:47.526283+0000","last_peered":"2024-09-16T01:10:47.526283+0000","last_clean":"2024-09-16T01:10:47.526283+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:10:47.526283+0000","last_undegraded":"2024-09-16T01:10:47.526283+0000","last_fullsized":"2024-09-16T01:10:47.526283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198235+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:10:47.198235+0000","last_peered":"2024-09-16T01:10:47.198235+0000","last_clean":"2024-09-16T01:10:47.198235+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:10:47.198235+0000","last_undegraded":"2024-09-16T01:10:47.198235+0000","last_fullsized":"2024-09-16T01:10:47.198235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525633+0000","last_change":"2024-09-16T01:10:40.108958+0000","last_active":"2024-09-16T01:10:47.525633+0000","last_peered":"2024-09-16T01:10:47.525633+0000","last_clean":"2024-09-16T01:10:47.525633+0000","last_became_active":"2024-09-16T01:10:40.099865+0000","last_became_peered":"2024-09-16T01:10:40.099865+0000","last_unstale":"2024-09-16T01:10:47.525633+0000","last_undegraded":"2024-09-16T01:10:47.525633+0000","last_fullsized":"2024-09-16T01:10:47.525633+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:48.312221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438311+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:10:47.438311+0000","last_peered":"2024-09-16T01:10:47.438311+0000","last_clean":"2024-09-16T01:10:47.438311+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:10:47.438311+0000","last_undegraded":"2024-09-16T01:10:47.438311+0000","last_fullsized":"2024-09-16T01:10:47.438311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198398+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:10:47.198398+0000","last_peered":"2024-09-16T01:10:47.198398+0000","last_clean":"2024-09-16T01:10:47.198398+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:10:47.198398+0000","last_undegraded":"2024-09-16T01:10:47.198398+0000","last_fullsized":"2024-09-16T01:10:47.198398+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290079+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:10:47.290079+0000","last_peered":"2024-09-16T01:10:47.290079+0000","last_clean":"2024-09-16T01:10:47.290079+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:10:47.290079+0000","last_undegraded":"2024-09-16T01:10:47.290079+0000","last_fullsized":"2024-09-16T01:10:47.290079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269004+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:10:47.269004+0000","last_peered":"2024-09-16T01:10:47.269004+0000","last_clean":"2024-09-16T01:10:47.269004+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:10:47.269004+0000","last_undegraded":"2024-09-16T01:10:47.269004+0000","last_fullsized":"2024-09-16T01:10:47.269004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198272+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:10:47.198272+0000","last_peered":"2024-09-16T01:10:47.198272+0000","last_clean":"2024-09-16T01:10:47.198272+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:10:47.198272+0000","last_undegraded":"2024-09-16T01:10:47.198272+0000","last_fullsized":"2024-09-16T01:10:47.198272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442368+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:10:47.442368+0000","last_peered":"2024-09-16T01:10:47.442368+0000","last_clean":"2024-09-16T01:10:47.442368+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:10:47.442368+0000","last_undegraded":"2024-09-16T01:10:47.442368+0000","last_fullsized":"2024-09-16T01:10:47.442368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289720+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:10:47.289720+0000","last_peered":"2024-09-16T01:10:47.289720+0000","last_clean":"2024-09-16T01:10:47.289720+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:10:47.289720+0000","last_undegraded":"2024-09-16T01:10:47.289720+0000","last_fullsized":"2024-09-16T01:10:47.289720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198536+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:10:47.198536+0000","last_peered":"2024-09-16T01:10:47.198536+0000","last_clean":"2024-09-16T01:10:47.198536+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:10:47.198536+0000","last_undegraded":"2024-09-16T01:10:47.198536+0000","last_fullsized":"2024-09-16T01:10:47.198536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198351+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:10:47.198351+0000","last_peered":"2024-09-16T01:10:47.198351+0000","last_clean":"2024-09-16T01:10:47.198351+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:10:47.198351+0000","last_undegraded":"2024-09-16T01:10:47.198351+0000","last_fullsized":"2024-09-16T01:10:47.198351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201616+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:10:47.201616+0000","last_peered":"2024-09-16T01:10:47.201616+0000","last_clean":"2024-09-16T01:10:47.201616+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:10:47.201616+0000","last_undegraded":"2024-09-16T01:10:47.201616+0000","last_fullsized":"2024-09-16T01:10:47.201616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198574+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:10:47.198574+0000","last_peered":"2024-09-16T01:10:47.198574+0000","last_clean":"2024-09-16T01:10:47.198574+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:10:47.198574+0000","last_undegraded":"2024-09-16T01:10:47.198574+0000","last_fullsized":"2024-09-16T01:10:47.198574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525563+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:10:47.525563+0000","last_peered":"2024-09-16T01:10:47.525563+0000","last_clean":"2024-09-16T01:10:47.525563+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:10:47.525563+0000","last_undegraded":"2024-09-16T01:10:47.525563+0000","last_fullsized":"2024-09-16T01:10:47.525563+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198104+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:10:47.198104+0000","last_peered":"2024-09-16T01:10:47.198104+0000","last_clean":"2024-09-16T01:10:47.198104+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:10:47.198104+0000","last_undegraded":"2024-09-16T01:10:47.198104+0000","last_fullsized":"2024-09-16T01:10:47.198104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287200+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:10:47.287200+0000","last_peered":"2024-09-16T01:10:47.287200+0000","last_clean":"2024-09-16T01:10:47.287200+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:10:47.287200+0000","last_undegraded":"2024-09-16T01:10:47.287200+0000","last_fullsized":"2024-09-16T01:10:47.287200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289566+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:10:47.289566+0000","last_peered":"2024-09-16T01:10:47.289566+0000","last_clean":"2024-09-16T01:10:47.289566+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:10:47.289566+0000","last_undegraded":"2024-09-16T01:10:47.289566+0000","last_fullsized":"2024-09-16T01:10:47.289566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438269+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:10:47.438269+0000","last_peered":"2024-09-16T01:10:47.438269+0000","last_clean":"2024-09-16T01:10:47.438269+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:10:47.438269+0000","last_undegraded":"2024-09-16T01:10:47.438269+0000","last_fullsized":"2024-09-16T01:10:47.438269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198667+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:10:47.198667+0000","last_peered":"2024-09-16T01:10:47.198667+0000","last_clean":"2024-09-16T01:10:47.198667+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:10:47.198667+0000","last_undegraded":"2024-09-16T01:10:47.198667+0000","last_fullsized":"2024-09-16T01:10:47.198667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526265+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:10:47.526265+0000","last_peered":"2024-09-16T01:10:47.526265+0000","last_clean":"2024-09-16T01:10:47.526265+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:10:47.526265+0000","last_undegraded":"2024-09-16T01:10:47.526265+0000","last_fullsized":"2024-09-16T01:10:47.526265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198806+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:10:47.198806+0000","last_peered":"2024-09-16T01:10:47.198806+0000","last_clean":"2024-09-16T01:10:47.198806+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:10:47.198806+0000","last_undegraded":"2024-09-16T01:10:47.198806+0000","last_fullsized":"2024-09-16T01:10:47.198806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201469+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:10:47.201469+0000","last_peered":"2024-09-16T01:10:47.201469+0000","last_clean":"2024-09-16T01:10:47.201469+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:10:47.201469+0000","last_undegraded":"2024-09-16T01:10:47.201469+0000","last_fullsized":"2024-09-16T01:10:47.201469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198893+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:10:47.198893+0000","last_peered":"2024-09-16T01:10:47.198893+0000","last_clean":"2024-09-16T01:10:47.198893+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:10:47.198893+0000","last_undegraded":"2024-09-16T01:10:47.198893+0000","last_fullsized":"2024-09-16T01:10:47.198893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'50","reported_seq":114,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.403760+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:10:47.403760+0000","last_peered":"2024-09-16T01:10:47.403760+0000","last_clean":"2024-09-16T01:10:47.403760+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:10:47.403760+0000","last_undegraded":"2024-09-16T01:10:47.403760+0000","last_fullsized":"2024-09-16T01:10:47.403760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":90,"num_write_kb":840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197770+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:10:47.197770+0000","last_peered":"2024-09-16T01:10:47.197770+0000","last_clean":"2024-09-16T01:10:47.197770+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:10:47.197770+0000","last_undegraded":"2024-09-16T01:10:47.197770+0000","last_fullsized":"2024-09-16T01:10:47.197770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271664+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:10:47.271664+0000","last_peered":"2024-09-16T01:10:47.271664+0000","last_clean":"2024-09-16T01:10:47.271664+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:10:47.271664+0000","last_undegraded":"2024-09-16T01:10:47.271664+0000","last_fullsized":"2024-09-16T01:10:47.271664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290000+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:10:47.290000+0000","last_peered":"2024-09-16T01:10:47.290000+0000","last_clean":"2024-09-16T01:10:47.290000+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:10:47.290000+0000","last_undegraded":"2024-09-16T01:10:47.290000+0000","last_fullsized":"2024-09-16T01:10:47.290000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525655+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:10:47.525655+0000","last_peered":"2024-09-16T01:10:47.525655+0000","last_clean":"2024-09-16T01:10:47.525655+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:10:47.525655+0000","last_undegraded":"2024-09-16T01:10:47.525655+0000","last_fullsized":"2024-09-16T01:10:47.525655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266701+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:10:47.266701+0000","last_peered":"2024-09-16T01:10:47.266701+0000","last_clean":"2024-09-16T01:10:47.266701+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:10:47.266701+0000","last_undegraded":"2024-09-16T01:10:47.266701+0000","last_fullsized":"2024-09-16T01:10:47.266701+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286991+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:10:47.286991+0000","last_peered":"2024-09-16T01:10:47.286991+0000","last_clean":"2024-09-16T01:10:47.286991+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:10:47.286991+0000","last_undegraded":"2024-09-16T01:10:47.286991+0000","last_fullsized":"2024-09-16T01:10:47.286991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289584+0000","last_change":"2024-09-16T01:09:39.117797+0000","last_active":"2024-09-16T01:10:47.289584+0000","last_peered":"2024-09-16T01:10:47.289584+0000","last_clean":"2024-09-16T01:10:47.289584+0000","last_became_active":"2024-09-16T01:09:39.117345+0000","last_became_peered":"2024-09-16T01:09:39.117345+0000","last_unstale":"2024-09-16T01:10:47.289584+0000","last_undegraded":"2024-09-16T01:10:47.289584+0000","last_fullsized":"2024-09-16T01:10:47.289584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:21.410065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266431+0000","last_change":"2024-09-16T01:09:39.148653+0000","last_active":"2024-09-16T01:10:47.266431+0000","last_peered":"2024-09-16T01:10:47.266431+0000","last_clean":"2024-09-16T01:10:47.266431+0000","last_became_active":"2024-09-16T01:09:39.148312+0000","last_became_peered":"2024-09-16T01:09:39.148312+0000","last_unstale":"2024-09-16T01:10:47.266431+0000","last_undegraded":"2024-09-16T01:10:47.266431+0000","last_fullsized":"2024-09-16T01:10:47.266431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:17.031741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442537+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:10:47.442537+0000","last_peered":"2024-09-16T01:10:47.442537+0000","last_clean":"2024-09-16T01:10:47.442537+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:10:47.442537+0000","last_undegraded":"2024-09-16T01:10:47.442537+0000","last_fullsized":"2024-09-16T01:10:47.442537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271403+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:10:47.271403+0000","last_peered":"2024-09-16T01:10:47.271403+0000","last_clean":"2024-09-16T01:10:47.271403+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:10:47.271403+0000","last_undegraded":"2024-09-16T01:10:47.271403+0000","last_fullsized":"2024-09-16T01:10:47.271403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268883+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:10:47.268883+0000","last_peered":"2024-09-16T01:10:47.268883+0000","last_clean":"2024-09-16T01:10:47.268883+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:10:47.268883+0000","last_undegraded":"2024-09-16T01:10:47.268883+0000","last_fullsized":"2024-09-16T01:10:47.268883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":148,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271283+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:10:47.271283+0000","last_peered":"2024-09-16T01:10:47.271283+0000","last_clean":"2024-09-16T01:10:47.271283+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:10:47.271283+0000","last_undegraded":"2024-09-16T01:10:47.271283+0000","last_fullsized":"2024-09-16T01:10:47.271283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272128+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:10:47.272128+0000","last_peered":"2024-09-16T01:10:47.272128+0000","last_clean":"2024-09-16T01:10:47.272128+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:10:47.272128+0000","last_undegraded":"2024-09-16T01:10:47.272128+0000","last_fullsized":"2024-09-16T01:10:47.272128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271097+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:10:47.271097+0000","last_peered":"2024-09-16T01:10:47.271097+0000","last_clean":"2024-09-16T01:10:47.271097+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:10:47.271097+0000","last_undegraded":"2024-09-16T01:10:47.271097+0000","last_fullsized":"2024-09-16T01:10:47.271097+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441989+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:10:47.441989+0000","last_peered":"2024-09-16T01:10:47.441989+0000","last_clean":"2024-09-16T01:10:47.441989+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:10:47.441989+0000","last_undegraded":"2024-09-16T01:10:47.441989+0000","last_fullsized":"2024-09-16T01:10:47.441989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269086+0000","last_change":"2024-09-16T01:09:39.167998+0000","last_active":"2024-09-16T01:10:47.269086+0000","last_peered":"2024-09-16T01:10:47.269086+0000","last_clean":"2024-09-16T01:10:47.269086+0000","last_became_active":"2024-09-16T01:09:39.167533+0000","last_became_peered":"2024-09-16T01:09:39.167533+0000","last_unstale":"2024-09-16T01:10:47.269086+0000","last_undegraded":"2024-09-16T01:10:47.269086+0000","last_fullsized":"2024-09-16T01:10:47.269086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:01.974490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198237+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:10:47.198237+0000","last_peered":"2024-09-16T01:10:47.198237+0000","last_clean":"2024-09-16T01:10:47.198237+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:10:47.198237+0000","last_undegraded":"2024-09-16T01:10:47.198237+0000","last_fullsized":"2024-09-16T01:10:47.198237+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198632+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:10:47.198632+0000","last_peered":"2024-09-16T01:10:47.198632+0000","last_clean":"2024-09-16T01:10:47.198632+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:10:47.198632+0000","last_undegraded":"2024-09-16T01:10:47.198632+0000","last_fullsized":"2024-09-16T01:10:47.198632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442104+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:10:47.442104+0000","last_peered":"2024-09-16T01:10:47.442104+0000","last_clean":"2024-09-16T01:10:47.442104+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:10:47.442104+0000","last_undegraded":"2024-09-16T01:10:47.442104+0000","last_fullsized":"2024-09-16T01:10:47.442104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442226+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:10:47.442226+0000","last_peered":"2024-09-16T01:10:47.442226+0000","last_clean":"2024-09-16T01:10:47.442226+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:10:47.442226+0000","last_undegraded":"2024-09-16T01:10:47.442226+0000","last_fullsized":"2024-09-16T01:10:47.442226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442414+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:10:47.442414+0000","last_peered":"2024-09-16T01:10:47.442414+0000","last_clean":"2024-09-16T01:10:47.442414+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:10:47.442414+0000","last_undegraded":"2024-09-16T01:10:47.442414+0000","last_fullsized":"2024-09-16T01:10:47.442414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201739+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:10:47.201739+0000","last_peered":"2024-09-16T01:10:47.201739+0000","last_clean":"2024-09-16T01:10:47.201739+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:10:47.201739+0000","last_undegraded":"2024-09-16T01:10:47.201739+0000","last_fullsized":"2024-09-16T01:10:47.201739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438356+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:10:47.438356+0000","last_peered":"2024-09-16T01:10:47.438356+0000","last_clean":"2024-09-16T01:10:47.438356+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:10:47.438356+0000","last_undegraded":"2024-09-16T01:10:47.438356+0000","last_fullsized":"2024-09-16T01:10:47.438356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442238+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:10:47.442238+0000","last_peered":"2024-09-16T01:10:47.442238+0000","last_clean":"2024-09-16T01:10:47.442238+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:10:47.442238+0000","last_undegraded":"2024-09-16T01:10:47.442238+0000","last_fullsized":"2024-09-16T01:10:47.442238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.396472+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:10:47.396472+0000","last_peered":"2024-09-16T01:10:47.396472+0000","last_clean":"2024-09-16T01:10:47.396472+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:10:47.396472+0000","last_undegraded":"2024-09-16T01:10:47.396472+0000","last_fullsized":"2024-09-16T01:10:47.396472+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525837+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:10:47.525837+0000","last_peered":"2024-09-16T01:10:47.525837+0000","last_clean":"2024-09-16T01:10:47.525837+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:10:47.525837+0000","last_undegraded":"2024-09-16T01:10:47.525837+0000","last_fullsized":"2024-09-16T01:10:47.525837+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442274+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:10:47.442274+0000","last_peered":"2024-09-16T01:10:47.442274+0000","last_clean":"2024-09-16T01:10:47.442274+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:10:47.442274+0000","last_undegraded":"2024-09-16T01:10:47.442274+0000","last_fullsized":"2024-09-16T01:10:47.442274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272022+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:10:47.272022+0000","last_peered":"2024-09-16T01:10:47.272022+0000","last_clean":"2024-09-16T01:10:47.272022+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:10:47.272022+0000","last_undegraded":"2024-09-16T01:10:47.272022+0000","last_fullsized":"2024-09-16T01:10:47.272022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268804+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:10:47.268804+0000","last_peered":"2024-09-16T01:10:47.268804+0000","last_clean":"2024-09-16T01:10:47.268804+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:10:47.268804+0000","last_undegraded":"2024-09-16T01:10:47.268804+0000","last_fullsized":"2024-09-16T01:10:47.268804+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525871+0000","last_change":"2024-09-16T01:09:39.119140+0000","last_active":"2024-09-16T01:10:47.525871+0000","last_peered":"2024-09-16T01:10:47.525871+0000","last_clean":"2024-09-16T01:10:47.525871+0000","last_became_active":"2024-09-16T01:09:39.118679+0000","last_became_peered":"2024-09-16T01:09:39.118679+0000","last_unstale":"2024-09-16T01:10:47.525871+0000","last_undegraded":"2024-09-16T01:10:47.525871+0000","last_fullsized":"2024-09-16T01:10:47.525871+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:14.355461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289913+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:10:47.289913+0000","last_peered":"2024-09-16T01:10:47.289913+0000","last_clean":"2024-09-16T01:10:47.289913+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:10:47.289913+0000","last_undegraded":"2024-09-16T01:10:47.289913+0000","last_fullsized":"2024-09-16T01:10:47.289913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268774+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:10:47.268774+0000","last_peered":"2024-09-16T01:10:47.268774+0000","last_clean":"2024-09-16T01:10:47.268774+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:10:47.268774+0000","last_undegraded":"2024-09-16T01:10:47.268774+0000","last_fullsized":"2024-09-16T01:10:47.268774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268875+0000","last_change":"2024-09-16T01:09:39.121178+0000","last_active":"2024-09-16T01:10:47.268875+0000","last_peered":"2024-09-16T01:10:47.268875+0000","last_clean":"2024-09-16T01:10:47.268875+0000","last_became_active":"2024-09-16T01:09:39.120781+0000","last_became_peered":"2024-09-16T01:09:39.120781+0000","last_unstale":"2024-09-16T01:10:47.268875+0000","last_undegraded":"2024-09-16T01:10:47.268875+0000","last_fullsized":"2024-09-16T01:10:47.268875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:43.670325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289670+0000","last_change":"2024-09-16T01:10:40.274011+0000","last_active":"2024-09-16T01:10:47.289670+0000","last_peered":"2024-09-16T01:10:47.289670+0000","last_clean":"2024-09-16T01:10:47.289670+0000","last_became_active":"2024-09-16T01:10:40.262422+0000","last_became_peered":"2024-09-16T01:10:40.262422+0000","last_unstale":"2024-09-16T01:10:47.289670+0000","last_undegraded":"2024-09-16T01:10:47.289670+0000","last_fullsized":"2024-09-16T01:10:47.289670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:32.563816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271887+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:10:47.271887+0000","last_peered":"2024-09-16T01:10:47.271887+0000","last_clean":"2024-09-16T01:10:47.271887+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:10:47.271887+0000","last_undegraded":"2024-09-16T01:10:47.271887+0000","last_fullsized":"2024-09-16T01:10:47.271887+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441971+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:10:47.441971+0000","last_peered":"2024-09-16T01:10:47.441971+0000","last_clean":"2024-09-16T01:10:47.441971+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:10:47.441971+0000","last_undegraded":"2024-09-16T01:10:47.441971+0000","last_fullsized":"2024-09-16T01:10:47.441971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":155,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441680+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:10:47.441680+0000","last_peered":"2024-09-16T01:10:47.441680+0000","last_clean":"2024-09-16T01:10:47.441680+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:10:47.441680+0000","last_undegraded":"2024-09-16T01:10:47.441680+0000","last_fullsized":"2024-09-16T01:10:47.441680+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1823,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287358+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:10:47.287358+0000","last_peered":"2024-09-16T01:10:47.287358+0000","last_clean":"2024-09-16T01:10:47.287358+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:10:47.287358+0000","last_undegraded":"2024-09-16T01:10:47.287358+0000","last_fullsized":"2024-09-16T01:10:47.287358+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442154+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:10:47.442154+0000","last_peered":"2024-09-16T01:10:47.442154+0000","last_clean":"2024-09-16T01:10:47.442154+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:10:47.442154+0000","last_undegraded":"2024-09-16T01:10:47.442154+0000","last_fullsized":"2024-09-16T01:10:47.442154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268920+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:10:47.268920+0000","last_peered":"2024-09-16T01:10:47.268920+0000","last_clean":"2024-09-16T01:10:47.268920+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:10:47.268920+0000","last_undegraded":"2024-09-16T01:10:47.268920+0000","last_fullsized":"2024-09-16T01:10:47.268920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525644+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:10:47.525644+0000","last_peered":"2024-09-16T01:10:47.525644+0000","last_clean":"2024-09-16T01:10:47.525644+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:10:47.525644+0000","last_undegraded":"2024-09-16T01:10:47.525644+0000","last_fullsized":"2024-09-16T01:10:47.525644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271305+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:10:47.271305+0000","last_peered":"2024-09-16T01:10:47.271305+0000","last_clean":"2024-09-16T01:10:47.271305+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:10:47.271305+0000","last_undegraded":"2024-09-16T01:10:47.271305+0000","last_fullsized":"2024-09-16T01:10:47.271305+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525622+0000","last_change":"2024-09-16T01:10:40.105562+0000","last_active":"2024-09-16T01:10:47.525622+0000","last_peered":"2024-09-16T01:10:47.525622+0000","last_clean":"2024-09-16T01:10:47.525622+0000","last_became_active":"2024-09-16T01:10:40.097568+0000","last_became_peered":"2024-09-16T01:10:40.097568+0000","last_unstale":"2024-09-16T01:10:47.525622+0000","last_undegraded":"2024-09-16T01:10:47.525622+0000","last_fullsized":"2024-09-16T01:10:47.525622+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:18.523898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268993+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:10:47.268993+0000","last_peered":"2024-09-16T01:10:47.268993+0000","last_clean":"2024-09-16T01:10:47.268993+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:10:47.268993+0000","last_undegraded":"2024-09-16T01:10:47.268993+0000","last_fullsized":"2024-09-16T01:10:47.268993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289920+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:10:47.289920+0000","last_peered":"2024-09-16T01:10:47.289920+0000","last_clean":"2024-09-16T01:10:47.289920+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:10:47.289920+0000","last_undegraded":"2024-09-16T01:10:47.289920+0000","last_fullsized":"2024-09-16T01:10:47.289920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":294,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438764+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:10:47.438764+0000","last_peered":"2024-09-16T01:10:47.438764+0000","last_clean":"2024-09-16T01:10:47.438764+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:10:47.438764+0000","last_undegraded":"2024-09-16T01:10:47.438764+0000","last_fullsized":"2024-09-16T01:10:47.438764+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68792,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":215,"num_read_kb":169,"num_write":39,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286615+0000","last_change":"2024-09-16T01:09:39.121127+0000","last_active":"2024-09-16T01:10:47.286615+0000","last_peered":"2024-09-16T01:10:47.286615+0000","last_clean":"2024-09-16T01:10:47.286615+0000","last_became_active":"2024-09-16T01:09:39.120736+0000","last_became_peered":"2024-09-16T01:09:39.120736+0000","last_unstale":"2024-09-16T01:10:47.286615+0000","last_undegraded":"2024-09-16T01:10:47.286615+0000","last_fullsized":"2024-09-16T01:10:47.286615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:28.281410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438518+0000","last_change":"2024-09-16T01:09:39.168566+0000","last_active":"2024-09-16T01:10:47.438518+0000","last_peered":"2024-09-16T01:10:47.438518+0000","last_clean":"2024-09-16T01:10:47.438518+0000","last_became_active":"2024-09-16T01:09:39.168127+0000","last_became_peered":"2024-09-16T01:09:39.168127+0000","last_unstale":"2024-09-16T01:10:47.438518+0000","last_undegraded":"2024-09-16T01:10:47.438518+0000","last_fullsized":"2024-09-16T01:10:47.438518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:52.054331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269209+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:10:47.269209+0000","last_peered":"2024-09-16T01:10:47.269209+0000","last_clean":"2024-09-16T01:10:47.269209+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:10:47.269209+0000","last_undegraded":"2024-09-16T01:10:47.269209+0000","last_fullsized":"2024-09-16T01:10:47.269209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198541+0000","last_change":"2024-09-16T01:09:39.121335+0000","last_active":"2024-09-16T01:10:47.198541+0000","last_peered":"2024-09-16T01:10:47.198541+0000","last_clean":"2024-09-16T01:10:47.198541+0000","last_became_active":"2024-09-16T01:09:39.120738+0000","last_became_peered":"2024-09-16T01:09:39.120738+0000","last_unstale":"2024-09-16T01:10:47.198541+0000","last_undegraded":"2024-09-16T01:10:47.198541+0000","last_fullsized":"2024-09-16T01:10:47.198541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:09.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526309+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:10:47.526309+0000","last_peered":"2024-09-16T01:10:47.526309+0000","last_clean":"2024-09-16T01:10:47.526309+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:10:47.526309+0000","last_undegraded":"2024-09-16T01:10:47.526309+0000","last_fullsized":"2024-09-16T01:10:47.526309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442222+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:10:47.442222+0000","last_peered":"2024-09-16T01:10:47.442222+0000","last_clean":"2024-09-16T01:10:47.442222+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:10:47.442222+0000","last_undegraded":"2024-09-16T01:10:47.442222+0000","last_fullsized":"2024-09-16T01:10:47.442222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286303+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:10:47.286303+0000","last_peered":"2024-09-16T01:10:47.286303+0000","last_clean":"2024-09-16T01:10:47.286303+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:10:47.286303+0000","last_undegraded":"2024-09-16T01:10:47.286303+0000","last_fullsized":"2024-09-16T01:10:47.286303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438818+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:10:47.438818+0000","last_peered":"2024-09-16T01:10:47.438818+0000","last_clean":"2024-09-16T01:10:47.438818+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:10:47.438818+0000","last_undegraded":"2024-09-16T01:10:47.438818+0000","last_fullsized":"2024-09-16T01:10:47.438818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":146,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266885+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:10:47.266885+0000","last_peered":"2024-09-16T01:10:47.266885+0000","last_clean":"2024-09-16T01:10:47.266885+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:10:47.266885+0000","last_undegraded":"2024-09-16T01:10:47.266885+0000","last_fullsized":"2024-09-16T01:10:47.266885+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197915+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:10:47.197915+0000","last_peered":"2024-09-16T01:10:47.197915+0000","last_clean":"2024-09-16T01:10:47.197915+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:10:47.197915+0000","last_undegraded":"2024-09-16T01:10:47.197915+0000","last_fullsized":"2024-09-16T01:10:47.197915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526102+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:10:47.526102+0000","last_peered":"2024-09-16T01:10:47.526102+0000","last_clean":"2024-09-16T01:10:47.526102+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:10:47.526102+0000","last_undegraded":"2024-09-16T01:10:47.526102+0000","last_fullsized":"2024-09-16T01:10:47.526102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286560+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:10:47.286560+0000","last_peered":"2024-09-16T01:10:47.286560+0000","last_clean":"2024-09-16T01:10:47.286560+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:10:47.286560+0000","last_undegraded":"2024-09-16T01:10:47.286560+0000","last_fullsized":"2024-09-16T01:10:47.286560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269361+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:10:47.269361+0000","last_peered":"2024-09-16T01:10:47.269361+0000","last_clean":"2024-09-16T01:10:47.269361+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:10:47.269361+0000","last_undegraded":"2024-09-16T01:10:47.269361+0000","last_fullsized":"2024-09-16T01:10:47.269361+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525752+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:10:47.525752+0000","last_peered":"2024-09-16T01:10:47.525752+0000","last_clean":"2024-09-16T01:10:47.525752+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:10:47.525752+0000","last_undegraded":"2024-09-16T01:10:47.525752+0000","last_fullsized":"2024-09-16T01:10:47.525752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197897+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:10:47.197897+0000","last_peered":"2024-09-16T01:10:47.197897+0000","last_clean":"2024-09-16T01:10:47.197897+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:10:47.197897+0000","last_undegraded":"2024-09-16T01:10:47.197897+0000","last_fullsized":"2024-09-16T01:10:47.197897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271530+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:10:47.271530+0000","last_peered":"2024-09-16T01:10:47.271530+0000","last_clean":"2024-09-16T01:10:47.271530+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:10:47.271530+0000","last_undegraded":"2024-09-16T01:10:47.271530+0000","last_fullsized":"2024-09-16T01:10:47.271530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266595+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:10:47.266595+0000","last_peered":"2024-09-16T01:10:47.266595+0000","last_clean":"2024-09-16T01:10:47.266595+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:10:47.266595+0000","last_undegraded":"2024-09-16T01:10:47.266595+0000","last_fullsized":"2024-09-16T01:10:47.266595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269140+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:10:47.269140+0000","last_peered":"2024-09-16T01:10:47.269140+0000","last_clean":"2024-09-16T01:10:47.269140+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:10:47.269140+0000","last_undegraded":"2024-09-16T01:10:47.269140+0000","last_fullsized":"2024-09-16T01:10:47.269140+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201517+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:10:47.201517+0000","last_peered":"2024-09-16T01:10:47.201517+0000","last_clean":"2024-09-16T01:10:47.201517+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:10:47.201517+0000","last_undegraded":"2024-09-16T01:10:47.201517+0000","last_fullsized":"2024-09-16T01:10:47.201517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'6","reported_seq":150,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269492+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:10:47.269492+0000","last_peered":"2024-09-16T01:10:47.269492+0000","last_clean":"2024-09-16T01:10:47.269492+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:10:47.269492+0000","last_undegraded":"2024-09-16T01:10:47.269492+0000","last_fullsized":"2024-09-16T01:10:47.269492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289678+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:10:47.289678+0000","last_peered":"2024-09-16T01:10:47.289678+0000","last_clean":"2024-09-16T01:10:47.289678+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:10:47.289678+0000","last_undegraded":"2024-09-16T01:10:47.289678+0000","last_fullsized":"2024-09-16T01:10:47.289678+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201763+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:10:47.201763+0000","last_peered":"2024-09-16T01:10:47.201763+0000","last_clean":"2024-09-16T01:10:47.201763+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:10:47.201763+0000","last_undegraded":"2024-09-16T01:10:47.201763+0000","last_fullsized":"2024-09-16T01:10:47.201763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525500+0000","last_change":"2024-09-16T01:09:39.119041+0000","last_active":"2024-09-16T01:10:47.525500+0000","last_peered":"2024-09-16T01:10:47.525500+0000","last_clean":"2024-09-16T01:10:47.525500+0000","last_became_active":"2024-09-16T01:09:39.118531+0000","last_became_peered":"2024-09-16T01:09:39.118531+0000","last_unstale":"2024-09-16T01:10:47.525500+0000","last_undegraded":"2024-09-16T01:10:47.525500+0000","last_fullsized":"2024-09-16T01:10:47.525500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:32:18.412355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201485+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:10:47.201485+0000","last_peered":"2024-09-16T01:10:47.201485+0000","last_clean":"2024-09-16T01:10:47.201485+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:10:47.201485+0000","last_undegraded":"2024-09-16T01:10:47.201485+0000","last_fullsized":"2024-09-16T01:10:47.201485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201337+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:10:47.201337+0000","last_peered":"2024-09-16T01:10:47.201337+0000","last_clean":"2024-09-16T01:10:47.201337+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:10:47.201337+0000","last_undegraded":"2024-09-16T01:10:47.201337+0000","last_fullsized":"2024-09-16T01:10:47.201337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525902+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:10:47.525902+0000","last_peered":"2024-09-16T01:10:47.525902+0000","last_clean":"2024-09-16T01:10:47.525902+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:10:47.525902+0000","last_undegraded":"2024-09-16T01:10:47.525902+0000","last_fullsized":"2024-09-16T01:10:47.525902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271674+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:10:47.271674+0000","last_peered":"2024-09-16T01:10:47.271674+0000","last_clean":"2024-09-16T01:10:47.271674+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:10:47.271674+0000","last_undegraded":"2024-09-16T01:10:47.271674+0000","last_fullsized":"2024-09-16T01:10:47.271674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525941+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:10:47.525941+0000","last_peered":"2024-09-16T01:10:47.525941+0000","last_clean":"2024-09-16T01:10:47.525941+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:10:47.525941+0000","last_undegraded":"2024-09-16T01:10:47.525941+0000","last_fullsized":"2024-09-16T01:10:47.525941+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201759+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:10:47.201759+0000","last_peered":"2024-09-16T01:10:47.201759+0000","last_clean":"2024-09-16T01:10:47.201759+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:10:47.201759+0000","last_undegraded":"2024-09-16T01:10:47.201759+0000","last_fullsized":"2024-09-16T01:10:47.201759+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198856+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:10:47.198856+0000","last_peered":"2024-09-16T01:10:47.198856+0000","last_clean":"2024-09-16T01:10:47.198856+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:10:47.198856+0000","last_undegraded":"2024-09-16T01:10:47.198856+0000","last_fullsized":"2024-09-16T01:10:47.198856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268742+0000","last_change":"2024-09-16T01:09:39.121080+0000","last_active":"2024-09-16T01:10:47.268742+0000","last_peered":"2024-09-16T01:10:47.268742+0000","last_clean":"2024-09-16T01:10:47.268742+0000","last_became_active":"2024-09-16T01:09:39.120707+0000","last_became_peered":"2024-09-16T01:09:39.120707+0000","last_unstale":"2024-09-16T01:10:47.268742+0000","last_undegraded":"2024-09-16T01:10:47.268742+0000","last_fullsized":"2024-09-16T01:10:47.268742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:08.521558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198060+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:10:47.198060+0000","last_peered":"2024-09-16T01:10:47.198060+0000","last_clean":"2024-09-16T01:10:47.198060+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:10:47.198060+0000","last_undegraded":"2024-09-16T01:10:47.198060+0000","last_fullsized":"2024-09-16T01:10:47.198060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438556+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:10:47.438556+0000","last_peered":"2024-09-16T01:10:47.438556+0000","last_clean":"2024-09-16T01:10:47.438556+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:10:47.438556+0000","last_undegraded":"2024-09-16T01:10:47.438556+0000","last_fullsized":"2024-09-16T01:10:47.438556+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271474+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:10:47.271474+0000","last_peered":"2024-09-16T01:10:47.271474+0000","last_clean":"2024-09-16T01:10:47.271474+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:10:47.271474+0000","last_undegraded":"2024-09-16T01:10:47.271474+0000","last_fullsized":"2024-09-16T01:10:47.271474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525822+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:10:47.525822+0000","last_peered":"2024-09-16T01:10:47.525822+0000","last_clean":"2024-09-16T01:10:47.525822+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:10:47.525822+0000","last_undegraded":"2024-09-16T01:10:47.525822+0000","last_fullsized":"2024-09-16T01:10:47.525822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201931+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:10:47.201931+0000","last_peered":"2024-09-16T01:10:47.201931+0000","last_clean":"2024-09-16T01:10:47.201931+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:10:47.201931+0000","last_undegraded":"2024-09-16T01:10:47.201931+0000","last_fullsized":"2024-09-16T01:10:47.201931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287352+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:10:47.287352+0000","last_peered":"2024-09-16T01:10:47.287352+0000","last_clean":"2024-09-16T01:10:47.287352+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:10:47.287352+0000","last_undegraded":"2024-09-16T01:10:47.287352+0000","last_fullsized":"2024-09-16T01:10:47.287352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525972+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:10:47.525972+0000","last_peered":"2024-09-16T01:10:47.525972+0000","last_clean":"2024-09-16T01:10:47.525972+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:10:47.525972+0000","last_undegraded":"2024-09-16T01:10:47.525972+0000","last_fullsized":"2024-09-16T01:10:47.525972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438123+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:10:47.438123+0000","last_peered":"2024-09-16T01:10:47.438123+0000","last_clean":"2024-09-16T01:10:47.438123+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:10:47.438123+0000","last_undegraded":"2024-09-16T01:10:47.438123+0000","last_fullsized":"2024-09-16T01:10:47.438123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525665+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:10:47.525665+0000","last_peered":"2024-09-16T01:10:47.525665+0000","last_clean":"2024-09-16T01:10:47.525665+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:10:47.525665+0000","last_undegraded":"2024-09-16T01:10:47.525665+0000","last_fullsized":"2024-09-16T01:10:47.525665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525687+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:10:47.525687+0000","last_peered":"2024-09-16T01:10:47.525687+0000","last_clean":"2024-09-16T01:10:47.525687+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:10:47.525687+0000","last_undegraded":"2024-09-16T01:10:47.525687+0000","last_fullsized":"2024-09-16T01:10:47.525687+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271378+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:10:47.271378+0000","last_peered":"2024-09-16T01:10:47.271378+0000","last_clean":"2024-09-16T01:10:47.271378+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:10:47.271378+0000","last_undegraded":"2024-09-16T01:10:47.271378+0000","last_fullsized":"2024-09-16T01:10:47.271378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198648+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:10:47.198648+0000","last_peered":"2024-09-16T01:10:47.198648+0000","last_clean":"2024-09-16T01:10:47.198648+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:10:47.198648+0000","last_undegraded":"2024-09-16T01:10:47.198648+0000","last_fullsized":"2024-09-16T01:10:47.198648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442364+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:10:47.442364+0000","last_peered":"2024-09-16T01:10:47.442364+0000","last_clean":"2024-09-16T01:10:47.442364+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:10:47.442364+0000","last_undegraded":"2024-09-16T01:10:47.442364+0000","last_fullsized":"2024-09-16T01:10:47.442364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266750+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:10:47.266750+0000","last_peered":"2024-09-16T01:10:47.266750+0000","last_clean":"2024-09-16T01:10:47.266750+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:10:47.266750+0000","last_undegraded":"2024-09-16T01:10:47.266750+0000","last_fullsized":"2024-09-16T01:10:47.266750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.439065+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:10:47.439065+0000","last_peered":"2024-09-16T01:10:47.439065+0000","last_clean":"2024-09-16T01:10:47.439065+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:10:47.439065+0000","last_undegraded":"2024-09-16T01:10:47.439065+0000","last_fullsized":"2024-09-16T01:10:47.439065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197771+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:10:47.197771+0000","last_peered":"2024-09-16T01:10:47.197771+0000","last_clean":"2024-09-16T01:10:47.197771+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:10:47.197771+0000","last_undegraded":"2024-09-16T01:10:47.197771+0000","last_fullsized":"2024-09-16T01:10:47.197771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442000+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:10:47.442000+0000","last_peered":"2024-09-16T01:10:47.442000+0000","last_clean":"2024-09-16T01:10:47.442000+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:10:47.442000+0000","last_undegraded":"2024-09-16T01:10:47.442000+0000","last_fullsized":"2024-09-16T01:10:47.442000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266403+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:10:47.266403+0000","last_peered":"2024-09-16T01:10:47.266403+0000","last_clean":"2024-09-16T01:10:47.266403+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:10:47.266403+0000","last_undegraded":"2024-09-16T01:10:47.266403+0000","last_fullsized":"2024-09-16T01:10:47.266403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289667+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:10:47.289667+0000","last_peered":"2024-09-16T01:10:47.289667+0000","last_clean":"2024-09-16T01:10:47.289667+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:10:47.289667+0000","last_undegraded":"2024-09-16T01:10:47.289667+0000","last_fullsized":"2024-09-16T01:10:47.289667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198621+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:10:47.198621+0000","last_peered":"2024-09-16T01:10:47.198621+0000","last_clean":"2024-09-16T01:10:47.198621+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:10:47.198621+0000","last_undegraded":"2024-09-16T01:10:47.198621+0000","last_fullsized":"2024-09-16T01:10:47.198621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525772+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:10:47.525772+0000","last_peered":"2024-09-16T01:10:47.525772+0000","last_clean":"2024-09-16T01:10:47.525772+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:10:47.525772+0000","last_undegraded":"2024-09-16T01:10:47.525772+0000","last_fullsized":"2024-09-16T01:10:47.525772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526118+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:10:47.526118+0000","last_peered":"2024-09-16T01:10:47.526118+0000","last_clean":"2024-09-16T01:10:47.526118+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:10:47.526118+0000","last_undegraded":"2024-09-16T01:10:47.526118+0000","last_fullsized":"2024-09-16T01:10:47.526118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525943+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:10:47.525943+0000","last_peered":"2024-09-16T01:10:47.525943+0000","last_clean":"2024-09-16T01:10:47.525943+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:10:47.525943+0000","last_undegraded":"2024-09-16T01:10:47.525943+0000","last_fullsized":"2024-09-16T01:10:47.525943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598846,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":736,"num_read_kb":473,"num_write":298,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1796538,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107904,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481834,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1040384,"data_stored":537345,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710766,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712784,"statfs":{"total":95995035648,"available":95961890816,"internally_reserved":0,"allocated":1208320,"data_stored":1326206,"data_compressed":74515,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874291,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":934859,"data_compressed":51537,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103224,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":930683,"data_compressed":51537,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":1134613,"data_compressed":68359,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528385,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1032192,"data_stored":532723,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1130496,"data_stored":1260566,"data_compressed":74515,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":538720,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1044480,"data_stored":532917,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":603022,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1105920,"data_stored":601425,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1056768,"data_stored":534111,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133469,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203205,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:10:52.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:10:52.876 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:53 smithi078 ceph-mon[30937]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:53 smithi078 ceph-mon[30937]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:53 smithi078 ceph-mon[30937]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-09-16T01:10:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:53 smithi148 ceph-mon[35030]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:53 smithi148 ceph-mon[35030]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:53 smithi148 ceph-mon[35030]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.438+0000 7f23a372a640 1 -- 172.21.15.78:0/4045525885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c105f40 msgr2=0x7f239c1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.438+0000 7f23a372a640 1 --2- 172.21.15.78:0/4045525885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c105f40 0x7f239c1063d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2394009900 tx=0x7f239402a470 comp rx=0 tx=0).stop 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- 172.21.15.78:0/4045525885 shutdown_connections 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- 172.21.15.78:0/4045525885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c105f40 0x7f239c1063d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- 172.21.15.78:0/4045525885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c101450 0x7f239c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- 172.21.15.78:0/4045525885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c100b30 0x7f239c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- 172.21.15.78:0/4045525885 >> 172.21.15.78:0/4045525885 conn(0x7f239c0fc7a0 msgr2=0x7f239c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- 172.21.15.78:0/4045525885 shutdown_connections 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- 172.21.15.78:0/4045525885 wait complete. 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 Processor -- start 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- start start 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c100b30 0x7f239c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 0x7f239c19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f239c118560 con 0x7f239c101450 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f239c1183e0 con 0x7f239c100b30 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.439+0000 7f23a372a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f239c1186e0 con 0x7f239c105f40 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42372/0 (socket says 172.21.15.78:42372) 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 -- 172.21.15.78:0/3638023677 learned_addr learned my addr 172.21.15.78:0/3638023677 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a14c6640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c100b30 0x7f239c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a1cc7640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 0x7f239c19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 -- 172.21.15.78:0/3638023677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 msgr2=0x7f239c19e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 0x7f239c19e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 -- 172.21.15.78:0/3638023677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c100b30 msgr2=0x7f239c1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c100b30 0x7f239c1a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 -- 172.21.15.78:0/3638023677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c19efb0 con 0x7f239c101450 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a1cc7640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 0x7f239c19e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a0cc5640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f238c00d950 tx=0x7f238c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c017030 con 0x7f239c101450 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c19f2a0 con 0x7f239c101450 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238c004020 con 0x7f239c101450 2024-09-16T01:10:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c0047a0 con 0x7f239c101450 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.440+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c1ab4f0 con 0x7f239c101450 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.441+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c06b070 con 0x7f239c101450 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.442+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 24) ==== 138582+0+0 (secure 0 0 0) 0x7f238c0041d0 con 0x7f239c101450 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.443+0000 7f23927fc640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 0x7f23880a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.443+0000 7f23a14c6640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 0x7f23880a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.443+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f238c0cfca0 con 0x7f239c101450 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.444+0000 7f23a14c6640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 0x7f23880a5990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2398005fd0 tx=0x7f2398009680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.447+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f238c09c0a0 con 0x7f239c101450 2024-09-16T01:10:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:53 smithi193 ceph-mon[35701]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:10:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:53 smithi193 ceph-mon[35701]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:53 smithi193 ceph-mon[35701]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-09-16T01:10:53.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.611+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f239c105010 con 0x7f23880a34d0 2024-09-16T01:10:53.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.619+0000 7f23927fc640 1 -- 172.21.15.78:0/3638023677 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339729 (secure 0 0 0) 0x7f239c105010 con 0x7f23880a34d0 2024-09-16T01:10:53.624 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:53.626 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 msgr2=0x7f23880a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 0x7f23880a5990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2398005fd0 tx=0x7f2398009680 comp rx=0 tx=0).stop 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 msgr2=0x7f239c1a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f238c00d950 tx=0x7f238c00de10 comp rx=0 tx=0).stop 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 shutdown_connections 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a34d0 0x7f23880a5990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f239c105f40 0x7f239c19e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f239c101450 0x7f239c1a4bd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 --2- 172.21.15.78:0/3638023677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f239c100b30 0x7f239c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:53.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.622+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 >> 172.21.15.78:0/3638023677 conn(0x7f239c0fc7a0 msgr2=0x7f239c1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:53.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.623+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 shutdown_connections 2024-09-16T01:10:53.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:53.623+0000 7f23a372a640 1 -- 172.21.15.78:0/3638023677 wait complete. 2024-09-16T01:10:54.560 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-09-16T01:10:53.215822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389779,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1153,"num_read_kb":841,"num_write":613,"num_write_kb":5304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378720,"kb_used_data":12792,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563104,"statfs":{"total":1151940427776,"available":1151552618496,"internally_reserved":0,"allocated":13099008,"data_stored":9862812,"data_compressed":565435,"data_compressed_allocated":765952,"data_compressed_original":4251648,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200331,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":281,"num_read_kb":249,"num_write":132,"num_write_kb":921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.997384"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287576+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:10:47.287576+0000","last_peered":"2024-09-16T01:10:47.287576+0000","last_clean":"2024-09-16T01:10:47.287576+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:10:47.287576+0000","last_undegraded":"2024-09-16T01:10:47.287576+0000","last_fullsized":"2024-09-16T01:10:47.287576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198446+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:10:47.198446+0000","last_peered":"2024-09-16T01:10:47.198446+0000","last_clean":"2024-09-16T01:10:47.198446+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:10:47.198446+0000","last_undegraded":"2024-09-16T01:10:47.198446+0000","last_fullsized":"2024-09-16T01:10:47.198446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201248+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:10:47.201248+0000","last_peered":"2024-09-16T01:10:47.201248+0000","last_clean":"2024-09-16T01:10:47.201248+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:10:47.201248+0000","last_undegraded":"2024-09-16T01:10:47.201248+0000","last_fullsized":"2024-09-16T01:10:47.201248+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198687+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:10:47.198687+0000","last_peered":"2024-09-16T01:10:47.198687+0000","last_clean":"2024-09-16T01:10:47.198687+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:10:47.198687+0000","last_undegraded":"2024-09-16T01:10:47.198687+0000","last_fullsized":"2024-09-16T01:10:47.198687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526283+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:10:47.526283+0000","last_peered":"2024-09-16T01:10:47.526283+0000","last_clean":"2024-09-16T01:10:47.526283+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:10:47.526283+0000","last_undegraded":"2024-09-16T01:10:47.526283+0000","last_fullsized":"2024-09-16T01:10:47.526283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198235+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:10:47.198235+0000","last_peered":"2024-09-16T01:10:47.198235+0000","last_clean":"2024-09-16T01:10:47.198235+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:10:47.198235+0000","last_undegraded":"2024-09-16T01:10:47.198235+0000","last_fullsized":"2024-09-16T01:10:47.198235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525633+0000","last_change":"2024-09-16T01:10:40.108958+0000","last_active":"2024-09-16T01:10:47.525633+0000","last_peered":"2024-09-16T01:10:47.525633+0000","last_clean":"2024-09-16T01:10:47.525633+0000","last_became_active":"2024-09-16T01:10:40.099865+0000","last_became_peered":"2024-09-16T01:10:40.099865+0000","last_unstale":"2024-09-16T01:10:47.525633+0000","last_undegraded":"2024-09-16T01:10:47.525633+0000","last_fullsized":"2024-09-16T01:10:47.525633+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:48.312221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438311+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:10:47.438311+0000","last_peered":"2024-09-16T01:10:47.438311+0000","last_clean":"2024-09-16T01:10:47.438311+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:10:47.438311+0000","last_undegraded":"2024-09-16T01:10:47.438311+0000","last_fullsized":"2024-09-16T01:10:47.438311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198398+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:10:47.198398+0000","last_peered":"2024-09-16T01:10:47.198398+0000","last_clean":"2024-09-16T01:10:47.198398+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:10:47.198398+0000","last_undegraded":"2024-09-16T01:10:47.198398+0000","last_fullsized":"2024-09-16T01:10:47.198398+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290079+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:10:47.290079+0000","last_peered":"2024-09-16T01:10:47.290079+0000","last_clean":"2024-09-16T01:10:47.290079+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:10:47.290079+0000","last_undegraded":"2024-09-16T01:10:47.290079+0000","last_fullsized":"2024-09-16T01:10:47.290079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269004+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:10:47.269004+0000","last_peered":"2024-09-16T01:10:47.269004+0000","last_clean":"2024-09-16T01:10:47.269004+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:10:47.269004+0000","last_undegraded":"2024-09-16T01:10:47.269004+0000","last_fullsized":"2024-09-16T01:10:47.269004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198272+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:10:47.198272+0000","last_peered":"2024-09-16T01:10:47.198272+0000","last_clean":"2024-09-16T01:10:47.198272+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:10:47.198272+0000","last_undegraded":"2024-09-16T01:10:47.198272+0000","last_fullsized":"2024-09-16T01:10:47.198272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442368+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:10:47.442368+0000","last_peered":"2024-09-16T01:10:47.442368+0000","last_clean":"2024-09-16T01:10:47.442368+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:10:47.442368+0000","last_undegraded":"2024-09-16T01:10:47.442368+0000","last_fullsized":"2024-09-16T01:10:47.442368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289720+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:10:47.289720+0000","last_peered":"2024-09-16T01:10:47.289720+0000","last_clean":"2024-09-16T01:10:47.289720+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:10:47.289720+0000","last_undegraded":"2024-09-16T01:10:47.289720+0000","last_fullsized":"2024-09-16T01:10:47.289720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198767+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:10:47.198767+0000","last_peered":"2024-09-16T01:10:47.198767+0000","last_clean":"2024-09-16T01:10:47.198767+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:10:47.198767+0000","last_undegraded":"2024-09-16T01:10:47.198767+0000","last_fullsized":"2024-09-16T01:10:47.198767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198351+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:10:47.198351+0000","last_peered":"2024-09-16T01:10:47.198351+0000","last_clean":"2024-09-16T01:10:47.198351+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:10:47.198351+0000","last_undegraded":"2024-09-16T01:10:47.198351+0000","last_fullsized":"2024-09-16T01:10:47.198351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201616+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:10:47.201616+0000","last_peered":"2024-09-16T01:10:47.201616+0000","last_clean":"2024-09-16T01:10:47.201616+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:10:47.201616+0000","last_undegraded":"2024-09-16T01:10:47.201616+0000","last_fullsized":"2024-09-16T01:10:47.201616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198574+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:10:47.198574+0000","last_peered":"2024-09-16T01:10:47.198574+0000","last_clean":"2024-09-16T01:10:47.198574+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:10:47.198574+0000","last_undegraded":"2024-09-16T01:10:47.198574+0000","last_fullsized":"2024-09-16T01:10:47.198574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525563+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:10:47.525563+0000","last_peered":"2024-09-16T01:10:47.525563+0000","last_clean":"2024-09-16T01:10:47.525563+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:10:47.525563+0000","last_undegraded":"2024-09-16T01:10:47.525563+0000","last_fullsized":"2024-09-16T01:10:47.525563+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198104+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:10:47.198104+0000","last_peered":"2024-09-16T01:10:47.198104+0000","last_clean":"2024-09-16T01:10:47.198104+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:10:47.198104+0000","last_undegraded":"2024-09-16T01:10:47.198104+0000","last_fullsized":"2024-09-16T01:10:47.198104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287247+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:10:47.287247+0000","last_peered":"2024-09-16T01:10:47.287247+0000","last_clean":"2024-09-16T01:10:47.287247+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:10:47.287247+0000","last_undegraded":"2024-09-16T01:10:47.287247+0000","last_fullsized":"2024-09-16T01:10:47.287247+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289566+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:10:47.289566+0000","last_peered":"2024-09-16T01:10:47.289566+0000","last_clean":"2024-09-16T01:10:47.289566+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:10:47.289566+0000","last_undegraded":"2024-09-16T01:10:47.289566+0000","last_fullsized":"2024-09-16T01:10:47.289566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438269+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:10:47.438269+0000","last_peered":"2024-09-16T01:10:47.438269+0000","last_clean":"2024-09-16T01:10:47.438269+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:10:47.438269+0000","last_undegraded":"2024-09-16T01:10:47.438269+0000","last_fullsized":"2024-09-16T01:10:47.438269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198841+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:10:47.198841+0000","last_peered":"2024-09-16T01:10:47.198841+0000","last_clean":"2024-09-16T01:10:47.198841+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:10:47.198841+0000","last_undegraded":"2024-09-16T01:10:47.198841+0000","last_fullsized":"2024-09-16T01:10:47.198841+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526265+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:10:47.526265+0000","last_peered":"2024-09-16T01:10:47.526265+0000","last_clean":"2024-09-16T01:10:47.526265+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:10:47.526265+0000","last_undegraded":"2024-09-16T01:10:47.526265+0000","last_fullsized":"2024-09-16T01:10:47.526265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198806+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:10:47.198806+0000","last_peered":"2024-09-16T01:10:47.198806+0000","last_clean":"2024-09-16T01:10:47.198806+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:10:47.198806+0000","last_undegraded":"2024-09-16T01:10:47.198806+0000","last_fullsized":"2024-09-16T01:10:47.198806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201469+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:10:47.201469+0000","last_peered":"2024-09-16T01:10:47.201469+0000","last_clean":"2024-09-16T01:10:47.201469+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:10:47.201469+0000","last_undegraded":"2024-09-16T01:10:47.201469+0000","last_fullsized":"2024-09-16T01:10:47.201469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198893+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:10:47.198893+0000","last_peered":"2024-09-16T01:10:47.198893+0000","last_clean":"2024-09-16T01:10:47.198893+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:10:47.198893+0000","last_undegraded":"2024-09-16T01:10:47.198893+0000","last_fullsized":"2024-09-16T01:10:47.198893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'104","reported_seq":237,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:50.800752+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:10:50.800752+0000","last_peered":"2024-09-16T01:10:50.800752+0000","last_clean":"2024-09-16T01:10:50.800752+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:10:50.800752+0000","last_undegraded":"2024-09-16T01:10:50.800752+0000","last_fullsized":"2024-09-16T01:10:50.800752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197770+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:10:47.197770+0000","last_peered":"2024-09-16T01:10:47.197770+0000","last_clean":"2024-09-16T01:10:47.197770+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:10:47.197770+0000","last_undegraded":"2024-09-16T01:10:47.197770+0000","last_fullsized":"2024-09-16T01:10:47.197770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271664+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:10:47.271664+0000","last_peered":"2024-09-16T01:10:47.271664+0000","last_clean":"2024-09-16T01:10:47.271664+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:10:47.271664+0000","last_undegraded":"2024-09-16T01:10:47.271664+0000","last_fullsized":"2024-09-16T01:10:47.271664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290000+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:10:47.290000+0000","last_peered":"2024-09-16T01:10:47.290000+0000","last_clean":"2024-09-16T01:10:47.290000+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:10:47.290000+0000","last_undegraded":"2024-09-16T01:10:47.290000+0000","last_fullsized":"2024-09-16T01:10:47.290000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525655+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:10:47.525655+0000","last_peered":"2024-09-16T01:10:47.525655+0000","last_clean":"2024-09-16T01:10:47.525655+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:10:47.525655+0000","last_undegraded":"2024-09-16T01:10:47.525655+0000","last_fullsized":"2024-09-16T01:10:47.525655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266701+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:10:47.266701+0000","last_peered":"2024-09-16T01:10:47.266701+0000","last_clean":"2024-09-16T01:10:47.266701+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:10:47.266701+0000","last_undegraded":"2024-09-16T01:10:47.266701+0000","last_fullsized":"2024-09-16T01:10:47.266701+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286991+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:10:47.286991+0000","last_peered":"2024-09-16T01:10:47.286991+0000","last_clean":"2024-09-16T01:10:47.286991+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:10:47.286991+0000","last_undegraded":"2024-09-16T01:10:47.286991+0000","last_fullsized":"2024-09-16T01:10:47.286991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289584+0000","last_change":"2024-09-16T01:09:39.117797+0000","last_active":"2024-09-16T01:10:47.289584+0000","last_peered":"2024-09-16T01:10:47.289584+0000","last_clean":"2024-09-16T01:10:47.289584+0000","last_became_active":"2024-09-16T01:09:39.117345+0000","last_became_peered":"2024-09-16T01:09:39.117345+0000","last_unstale":"2024-09-16T01:10:47.289584+0000","last_undegraded":"2024-09-16T01:10:47.289584+0000","last_fullsized":"2024-09-16T01:10:47.289584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:21.410065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266431+0000","last_change":"2024-09-16T01:09:39.148653+0000","last_active":"2024-09-16T01:10:47.266431+0000","last_peered":"2024-09-16T01:10:47.266431+0000","last_clean":"2024-09-16T01:10:47.266431+0000","last_became_active":"2024-09-16T01:09:39.148312+0000","last_became_peered":"2024-09-16T01:09:39.148312+0000","last_unstale":"2024-09-16T01:10:47.266431+0000","last_undegraded":"2024-09-16T01:10:47.266431+0000","last_fullsized":"2024-09-16T01:10:47.266431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:17.031741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442537+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:10:47.442537+0000","last_peered":"2024-09-16T01:10:47.442537+0000","last_clean":"2024-09-16T01:10:47.442537+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:10:47.442537+0000","last_undegraded":"2024-09-16T01:10:47.442537+0000","last_fullsized":"2024-09-16T01:10:47.442537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271403+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:10:47.271403+0000","last_peered":"2024-09-16T01:10:47.271403+0000","last_clean":"2024-09-16T01:10:47.271403+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:10:47.271403+0000","last_undegraded":"2024-09-16T01:10:47.271403+0000","last_fullsized":"2024-09-16T01:10:47.271403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268883+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:10:47.268883+0000","last_peered":"2024-09-16T01:10:47.268883+0000","last_clean":"2024-09-16T01:10:47.268883+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:10:47.268883+0000","last_undegraded":"2024-09-16T01:10:47.268883+0000","last_fullsized":"2024-09-16T01:10:47.268883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":148,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271283+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:10:47.271283+0000","last_peered":"2024-09-16T01:10:47.271283+0000","last_clean":"2024-09-16T01:10:47.271283+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:10:47.271283+0000","last_undegraded":"2024-09-16T01:10:47.271283+0000","last_fullsized":"2024-09-16T01:10:47.271283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272128+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:10:47.272128+0000","last_peered":"2024-09-16T01:10:47.272128+0000","last_clean":"2024-09-16T01:10:47.272128+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:10:47.272128+0000","last_undegraded":"2024-09-16T01:10:47.272128+0000","last_fullsized":"2024-09-16T01:10:47.272128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271097+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:10:47.271097+0000","last_peered":"2024-09-16T01:10:47.271097+0000","last_clean":"2024-09-16T01:10:47.271097+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:10:47.271097+0000","last_undegraded":"2024-09-16T01:10:47.271097+0000","last_fullsized":"2024-09-16T01:10:47.271097+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441989+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:10:47.441989+0000","last_peered":"2024-09-16T01:10:47.441989+0000","last_clean":"2024-09-16T01:10:47.441989+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:10:47.441989+0000","last_undegraded":"2024-09-16T01:10:47.441989+0000","last_fullsized":"2024-09-16T01:10:47.441989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269086+0000","last_change":"2024-09-16T01:09:39.167998+0000","last_active":"2024-09-16T01:10:47.269086+0000","last_peered":"2024-09-16T01:10:47.269086+0000","last_clean":"2024-09-16T01:10:47.269086+0000","last_became_active":"2024-09-16T01:09:39.167533+0000","last_became_peered":"2024-09-16T01:09:39.167533+0000","last_unstale":"2024-09-16T01:10:47.269086+0000","last_undegraded":"2024-09-16T01:10:47.269086+0000","last_fullsized":"2024-09-16T01:10:47.269086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:01.974490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198237+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:10:47.198237+0000","last_peered":"2024-09-16T01:10:47.198237+0000","last_clean":"2024-09-16T01:10:47.198237+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:10:47.198237+0000","last_undegraded":"2024-09-16T01:10:47.198237+0000","last_fullsized":"2024-09-16T01:10:47.198237+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198632+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:10:47.198632+0000","last_peered":"2024-09-16T01:10:47.198632+0000","last_clean":"2024-09-16T01:10:47.198632+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:10:47.198632+0000","last_undegraded":"2024-09-16T01:10:47.198632+0000","last_fullsized":"2024-09-16T01:10:47.198632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442104+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:10:47.442104+0000","last_peered":"2024-09-16T01:10:47.442104+0000","last_clean":"2024-09-16T01:10:47.442104+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:10:47.442104+0000","last_undegraded":"2024-09-16T01:10:47.442104+0000","last_fullsized":"2024-09-16T01:10:47.442104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442226+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:10:47.442226+0000","last_peered":"2024-09-16T01:10:47.442226+0000","last_clean":"2024-09-16T01:10:47.442226+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:10:47.442226+0000","last_undegraded":"2024-09-16T01:10:47.442226+0000","last_fullsized":"2024-09-16T01:10:47.442226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442414+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:10:47.442414+0000","last_peered":"2024-09-16T01:10:47.442414+0000","last_clean":"2024-09-16T01:10:47.442414+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:10:47.442414+0000","last_undegraded":"2024-09-16T01:10:47.442414+0000","last_fullsized":"2024-09-16T01:10:47.442414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201739+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:10:47.201739+0000","last_peered":"2024-09-16T01:10:47.201739+0000","last_clean":"2024-09-16T01:10:47.201739+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:10:47.201739+0000","last_undegraded":"2024-09-16T01:10:47.201739+0000","last_fullsized":"2024-09-16T01:10:47.201739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438356+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:10:47.438356+0000","last_peered":"2024-09-16T01:10:47.438356+0000","last_clean":"2024-09-16T01:10:47.438356+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:10:47.438356+0000","last_undegraded":"2024-09-16T01:10:47.438356+0000","last_fullsized":"2024-09-16T01:10:47.438356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442238+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:10:47.442238+0000","last_peered":"2024-09-16T01:10:47.442238+0000","last_clean":"2024-09-16T01:10:47.442238+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:10:47.442238+0000","last_undegraded":"2024-09-16T01:10:47.442238+0000","last_fullsized":"2024-09-16T01:10:47.442238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":137,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:50.432697+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:10:50.432697+0000","last_peered":"2024-09-16T01:10:50.432697+0000","last_clean":"2024-09-16T01:10:50.432697+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:10:50.432697+0000","last_undegraded":"2024-09-16T01:10:50.432697+0000","last_fullsized":"2024-09-16T01:10:50.432697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525865+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:10:47.525865+0000","last_peered":"2024-09-16T01:10:47.525865+0000","last_clean":"2024-09-16T01:10:47.525865+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:10:47.525865+0000","last_undegraded":"2024-09-16T01:10:47.525865+0000","last_fullsized":"2024-09-16T01:10:47.525865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442274+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:10:47.442274+0000","last_peered":"2024-09-16T01:10:47.442274+0000","last_clean":"2024-09-16T01:10:47.442274+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:10:47.442274+0000","last_undegraded":"2024-09-16T01:10:47.442274+0000","last_fullsized":"2024-09-16T01:10:47.442274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272022+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:10:47.272022+0000","last_peered":"2024-09-16T01:10:47.272022+0000","last_clean":"2024-09-16T01:10:47.272022+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:10:47.272022+0000","last_undegraded":"2024-09-16T01:10:47.272022+0000","last_fullsized":"2024-09-16T01:10:47.272022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268804+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:10:47.268804+0000","last_peered":"2024-09-16T01:10:47.268804+0000","last_clean":"2024-09-16T01:10:47.268804+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:10:47.268804+0000","last_undegraded":"2024-09-16T01:10:47.268804+0000","last_fullsized":"2024-09-16T01:10:47.268804+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525871+0000","last_change":"2024-09-16T01:09:39.119140+0000","last_active":"2024-09-16T01:10:47.525871+0000","last_peered":"2024-09-16T01:10:47.525871+0000","last_clean":"2024-09-16T01:10:47.525871+0000","last_became_active":"2024-09-16T01:09:39.118679+0000","last_became_peered":"2024-09-16T01:09:39.118679+0000","last_unstale":"2024-09-16T01:10:47.525871+0000","last_undegraded":"2024-09-16T01:10:47.525871+0000","last_fullsized":"2024-09-16T01:10:47.525871+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:14.355461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289913+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:10:47.289913+0000","last_peered":"2024-09-16T01:10:47.289913+0000","last_clean":"2024-09-16T01:10:47.289913+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:10:47.289913+0000","last_undegraded":"2024-09-16T01:10:47.289913+0000","last_fullsized":"2024-09-16T01:10:47.289913+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268774+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:10:47.268774+0000","last_peered":"2024-09-16T01:10:47.268774+0000","last_clean":"2024-09-16T01:10:47.268774+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:10:47.268774+0000","last_undegraded":"2024-09-16T01:10:47.268774+0000","last_fullsized":"2024-09-16T01:10:47.268774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268924+0000","last_change":"2024-09-16T01:09:39.121178+0000","last_active":"2024-09-16T01:10:47.268924+0000","last_peered":"2024-09-16T01:10:47.268924+0000","last_clean":"2024-09-16T01:10:47.268924+0000","last_became_active":"2024-09-16T01:09:39.120781+0000","last_became_peered":"2024-09-16T01:09:39.120781+0000","last_unstale":"2024-09-16T01:10:47.268924+0000","last_undegraded":"2024-09-16T01:10:47.268924+0000","last_fullsized":"2024-09-16T01:10:47.268924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:43.670325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289670+0000","last_change":"2024-09-16T01:10:40.274011+0000","last_active":"2024-09-16T01:10:47.289670+0000","last_peered":"2024-09-16T01:10:47.289670+0000","last_clean":"2024-09-16T01:10:47.289670+0000","last_became_active":"2024-09-16T01:10:40.262422+0000","last_became_peered":"2024-09-16T01:10:40.262422+0000","last_unstale":"2024-09-16T01:10:47.289670+0000","last_undegraded":"2024-09-16T01:10:47.289670+0000","last_fullsized":"2024-09-16T01:10:47.289670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:32.563816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271887+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:10:47.271887+0000","last_peered":"2024-09-16T01:10:47.271887+0000","last_clean":"2024-09-16T01:10:47.271887+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:10:47.271887+0000","last_undegraded":"2024-09-16T01:10:47.271887+0000","last_fullsized":"2024-09-16T01:10:47.271887+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441971+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:10:47.441971+0000","last_peered":"2024-09-16T01:10:47.441971+0000","last_clean":"2024-09-16T01:10:47.441971+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:10:47.441971+0000","last_undegraded":"2024-09-16T01:10:47.441971+0000","last_fullsized":"2024-09-16T01:10:47.441971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:51.277292+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:10:51.277292+0000","last_peered":"2024-09-16T01:10:51.277292+0000","last_clean":"2024-09-16T01:10:51.277292+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:10:51.277292+0000","last_undegraded":"2024-09-16T01:10:51.277292+0000","last_fullsized":"2024-09-16T01:10:51.277292+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287358+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:10:47.287358+0000","last_peered":"2024-09-16T01:10:47.287358+0000","last_clean":"2024-09-16T01:10:47.287358+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:10:47.287358+0000","last_undegraded":"2024-09-16T01:10:47.287358+0000","last_fullsized":"2024-09-16T01:10:47.287358+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442154+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:10:47.442154+0000","last_peered":"2024-09-16T01:10:47.442154+0000","last_clean":"2024-09-16T01:10:47.442154+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:10:47.442154+0000","last_undegraded":"2024-09-16T01:10:47.442154+0000","last_fullsized":"2024-09-16T01:10:47.442154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268920+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:10:47.268920+0000","last_peered":"2024-09-16T01:10:47.268920+0000","last_clean":"2024-09-16T01:10:47.268920+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:10:47.268920+0000","last_undegraded":"2024-09-16T01:10:47.268920+0000","last_fullsized":"2024-09-16T01:10:47.268920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525644+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:10:47.525644+0000","last_peered":"2024-09-16T01:10:47.525644+0000","last_clean":"2024-09-16T01:10:47.525644+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:10:47.525644+0000","last_undegraded":"2024-09-16T01:10:47.525644+0000","last_fullsized":"2024-09-16T01:10:47.525644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271305+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:10:47.271305+0000","last_peered":"2024-09-16T01:10:47.271305+0000","last_clean":"2024-09-16T01:10:47.271305+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:10:47.271305+0000","last_undegraded":"2024-09-16T01:10:47.271305+0000","last_fullsized":"2024-09-16T01:10:47.271305+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525622+0000","last_change":"2024-09-16T01:10:40.105562+0000","last_active":"2024-09-16T01:10:47.525622+0000","last_peered":"2024-09-16T01:10:47.525622+0000","last_clean":"2024-09-16T01:10:47.525622+0000","last_became_active":"2024-09-16T01:10:40.097568+0000","last_became_peered":"2024-09-16T01:10:40.097568+0000","last_unstale":"2024-09-16T01:10:47.525622+0000","last_undegraded":"2024-09-16T01:10:47.525622+0000","last_fullsized":"2024-09-16T01:10:47.525622+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:18.523898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268993+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:10:47.268993+0000","last_peered":"2024-09-16T01:10:47.268993+0000","last_clean":"2024-09-16T01:10:47.268993+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:10:47.268993+0000","last_undegraded":"2024-09-16T01:10:47.268993+0000","last_fullsized":"2024-09-16T01:10:47.268993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289920+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:10:47.289920+0000","last_peered":"2024-09-16T01:10:47.289920+0000","last_clean":"2024-09-16T01:10:47.289920+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:10:47.289920+0000","last_undegraded":"2024-09-16T01:10:47.289920+0000","last_fullsized":"2024-09-16T01:10:47.289920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":307,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:52.275701+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:10:52.275701+0000","last_peered":"2024-09-16T01:10:52.275701+0000","last_clean":"2024-09-16T01:10:52.275701+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:10:52.275701+0000","last_undegraded":"2024-09-16T01:10:52.275701+0000","last_fullsized":"2024-09-16T01:10:52.275701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71705,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":181,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286615+0000","last_change":"2024-09-16T01:09:39.121127+0000","last_active":"2024-09-16T01:10:47.286615+0000","last_peered":"2024-09-16T01:10:47.286615+0000","last_clean":"2024-09-16T01:10:47.286615+0000","last_became_active":"2024-09-16T01:09:39.120736+0000","last_became_peered":"2024-09-16T01:09:39.120736+0000","last_unstale":"2024-09-16T01:10:47.286615+0000","last_undegraded":"2024-09-16T01:10:47.286615+0000","last_fullsized":"2024-09-16T01:10:47.286615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:28.281410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438518+0000","last_change":"2024-09-16T01:09:39.168566+0000","last_active":"2024-09-16T01:10:47.438518+0000","last_peered":"2024-09-16T01:10:47.438518+0000","last_clean":"2024-09-16T01:10:47.438518+0000","last_became_active":"2024-09-16T01:09:39.168127+0000","last_became_peered":"2024-09-16T01:09:39.168127+0000","last_unstale":"2024-09-16T01:10:47.438518+0000","last_undegraded":"2024-09-16T01:10:47.438518+0000","last_fullsized":"2024-09-16T01:10:47.438518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:52.054331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269209+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:10:47.269209+0000","last_peered":"2024-09-16T01:10:47.269209+0000","last_clean":"2024-09-16T01:10:47.269209+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:10:47.269209+0000","last_undegraded":"2024-09-16T01:10:47.269209+0000","last_fullsized":"2024-09-16T01:10:47.269209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198541+0000","last_change":"2024-09-16T01:09:39.121335+0000","last_active":"2024-09-16T01:10:47.198541+0000","last_peered":"2024-09-16T01:10:47.198541+0000","last_clean":"2024-09-16T01:10:47.198541+0000","last_became_active":"2024-09-16T01:09:39.120738+0000","last_became_peered":"2024-09-16T01:09:39.120738+0000","last_unstale":"2024-09-16T01:10:47.198541+0000","last_undegraded":"2024-09-16T01:10:47.198541+0000","last_fullsized":"2024-09-16T01:10:47.198541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:09.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526309+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:10:47.526309+0000","last_peered":"2024-09-16T01:10:47.526309+0000","last_clean":"2024-09-16T01:10:47.526309+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:10:47.526309+0000","last_undegraded":"2024-09-16T01:10:47.526309+0000","last_fullsized":"2024-09-16T01:10:47.526309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442222+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:10:47.442222+0000","last_peered":"2024-09-16T01:10:47.442222+0000","last_clean":"2024-09-16T01:10:47.442222+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:10:47.442222+0000","last_undegraded":"2024-09-16T01:10:47.442222+0000","last_fullsized":"2024-09-16T01:10:47.442222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286303+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:10:47.286303+0000","last_peered":"2024-09-16T01:10:47.286303+0000","last_clean":"2024-09-16T01:10:47.286303+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:10:47.286303+0000","last_undegraded":"2024-09-16T01:10:47.286303+0000","last_fullsized":"2024-09-16T01:10:47.286303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438818+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:10:47.438818+0000","last_peered":"2024-09-16T01:10:47.438818+0000","last_clean":"2024-09-16T01:10:47.438818+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:10:47.438818+0000","last_undegraded":"2024-09-16T01:10:47.438818+0000","last_fullsized":"2024-09-16T01:10:47.438818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:52.275173+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:10:52.275173+0000","last_peered":"2024-09-16T01:10:52.275173+0000","last_clean":"2024-09-16T01:10:52.275173+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:10:52.275173+0000","last_undegraded":"2024-09-16T01:10:52.275173+0000","last_fullsized":"2024-09-16T01:10:52.275173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197915+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:10:47.197915+0000","last_peered":"2024-09-16T01:10:47.197915+0000","last_clean":"2024-09-16T01:10:47.197915+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:10:47.197915+0000","last_undegraded":"2024-09-16T01:10:47.197915+0000","last_fullsized":"2024-09-16T01:10:47.197915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526102+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:10:47.526102+0000","last_peered":"2024-09-16T01:10:47.526102+0000","last_clean":"2024-09-16T01:10:47.526102+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:10:47.526102+0000","last_undegraded":"2024-09-16T01:10:47.526102+0000","last_fullsized":"2024-09-16T01:10:47.526102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286613+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:10:47.286613+0000","last_peered":"2024-09-16T01:10:47.286613+0000","last_clean":"2024-09-16T01:10:47.286613+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:10:47.286613+0000","last_undegraded":"2024-09-16T01:10:47.286613+0000","last_fullsized":"2024-09-16T01:10:47.286613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269391+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:10:47.269391+0000","last_peered":"2024-09-16T01:10:47.269391+0000","last_clean":"2024-09-16T01:10:47.269391+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:10:47.269391+0000","last_undegraded":"2024-09-16T01:10:47.269391+0000","last_fullsized":"2024-09-16T01:10:47.269391+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525752+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:10:47.525752+0000","last_peered":"2024-09-16T01:10:47.525752+0000","last_clean":"2024-09-16T01:10:47.525752+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:10:47.525752+0000","last_undegraded":"2024-09-16T01:10:47.525752+0000","last_fullsized":"2024-09-16T01:10:47.525752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197897+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:10:47.197897+0000","last_peered":"2024-09-16T01:10:47.197897+0000","last_clean":"2024-09-16T01:10:47.197897+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:10:47.197897+0000","last_undegraded":"2024-09-16T01:10:47.197897+0000","last_fullsized":"2024-09-16T01:10:47.197897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271530+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:10:47.271530+0000","last_peered":"2024-09-16T01:10:47.271530+0000","last_clean":"2024-09-16T01:10:47.271530+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:10:47.271530+0000","last_undegraded":"2024-09-16T01:10:47.271530+0000","last_fullsized":"2024-09-16T01:10:47.271530+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266595+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:10:47.266595+0000","last_peered":"2024-09-16T01:10:47.266595+0000","last_clean":"2024-09-16T01:10:47.266595+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:10:47.266595+0000","last_undegraded":"2024-09-16T01:10:47.266595+0000","last_fullsized":"2024-09-16T01:10:47.266595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269140+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:10:47.269140+0000","last_peered":"2024-09-16T01:10:47.269140+0000","last_clean":"2024-09-16T01:10:47.269140+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:10:47.269140+0000","last_undegraded":"2024-09-16T01:10:47.269140+0000","last_fullsized":"2024-09-16T01:10:47.269140+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201517+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:10:47.201517+0000","last_peered":"2024-09-16T01:10:47.201517+0000","last_clean":"2024-09-16T01:10:47.201517+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:10:47.201517+0000","last_undegraded":"2024-09-16T01:10:47.201517+0000","last_fullsized":"2024-09-16T01:10:47.201517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:51.274451+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:10:51.274451+0000","last_peered":"2024-09-16T01:10:51.274451+0000","last_clean":"2024-09-16T01:10:51.274451+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:10:51.274451+0000","last_undegraded":"2024-09-16T01:10:51.274451+0000","last_fullsized":"2024-09-16T01:10:51.274451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289678+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:10:47.289678+0000","last_peered":"2024-09-16T01:10:47.289678+0000","last_clean":"2024-09-16T01:10:47.289678+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:10:47.289678+0000","last_undegraded":"2024-09-16T01:10:47.289678+0000","last_fullsized":"2024-09-16T01:10:47.289678+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201763+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:10:47.201763+0000","last_peered":"2024-09-16T01:10:47.201763+0000","last_clean":"2024-09-16T01:10:47.201763+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:10:47.201763+0000","last_undegraded":"2024-09-16T01:10:47.201763+0000","last_fullsized":"2024-09-16T01:10:47.201763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525500+0000","last_change":"2024-09-16T01:09:39.119041+0000","last_active":"2024-09-16T01:10:47.525500+0000","last_peered":"2024-09-16T01:10:47.525500+0000","last_clean":"2024-09-16T01:10:47.525500+0000","last_became_active":"2024-09-16T01:09:39.118531+0000","last_became_peered":"2024-09-16T01:09:39.118531+0000","last_unstale":"2024-09-16T01:10:47.525500+0000","last_undegraded":"2024-09-16T01:10:47.525500+0000","last_fullsized":"2024-09-16T01:10:47.525500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:32:18.412355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201485+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:10:47.201485+0000","last_peered":"2024-09-16T01:10:47.201485+0000","last_clean":"2024-09-16T01:10:47.201485+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:10:47.201485+0000","last_undegraded":"2024-09-16T01:10:47.201485+0000","last_fullsized":"2024-09-16T01:10:47.201485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201337+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:10:47.201337+0000","last_peered":"2024-09-16T01:10:47.201337+0000","last_clean":"2024-09-16T01:10:47.201337+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:10:47.201337+0000","last_undegraded":"2024-09-16T01:10:47.201337+0000","last_fullsized":"2024-09-16T01:10:47.201337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525902+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:10:47.525902+0000","last_peered":"2024-09-16T01:10:47.525902+0000","last_clean":"2024-09-16T01:10:47.525902+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:10:47.525902+0000","last_undegraded":"2024-09-16T01:10:47.525902+0000","last_fullsized":"2024-09-16T01:10:47.525902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271674+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:10:47.271674+0000","last_peered":"2024-09-16T01:10:47.271674+0000","last_clean":"2024-09-16T01:10:47.271674+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:10:47.271674+0000","last_undegraded":"2024-09-16T01:10:47.271674+0000","last_fullsized":"2024-09-16T01:10:47.271674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525941+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:10:47.525941+0000","last_peered":"2024-09-16T01:10:47.525941+0000","last_clean":"2024-09-16T01:10:47.525941+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:10:47.525941+0000","last_undegraded":"2024-09-16T01:10:47.525941+0000","last_fullsized":"2024-09-16T01:10:47.525941+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201759+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:10:47.201759+0000","last_peered":"2024-09-16T01:10:47.201759+0000","last_clean":"2024-09-16T01:10:47.201759+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:10:47.201759+0000","last_undegraded":"2024-09-16T01:10:47.201759+0000","last_fullsized":"2024-09-16T01:10:47.201759+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198856+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:10:47.198856+0000","last_peered":"2024-09-16T01:10:47.198856+0000","last_clean":"2024-09-16T01:10:47.198856+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:10:47.198856+0000","last_undegraded":"2024-09-16T01:10:47.198856+0000","last_fullsized":"2024-09-16T01:10:47.198856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268742+0000","last_change":"2024-09-16T01:09:39.121080+0000","last_active":"2024-09-16T01:10:47.268742+0000","last_peered":"2024-09-16T01:10:47.268742+0000","last_clean":"2024-09-16T01:10:47.268742+0000","last_became_active":"2024-09-16T01:09:39.120707+0000","last_became_peered":"2024-09-16T01:09:39.120707+0000","last_unstale":"2024-09-16T01:10:47.268742+0000","last_undegraded":"2024-09-16T01:10:47.268742+0000","last_fullsized":"2024-09-16T01:10:47.268742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:08.521558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198060+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:10:47.198060+0000","last_peered":"2024-09-16T01:10:47.198060+0000","last_clean":"2024-09-16T01:10:47.198060+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:10:47.198060+0000","last_undegraded":"2024-09-16T01:10:47.198060+0000","last_fullsized":"2024-09-16T01:10:47.198060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438556+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:10:47.438556+0000","last_peered":"2024-09-16T01:10:47.438556+0000","last_clean":"2024-09-16T01:10:47.438556+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:10:47.438556+0000","last_undegraded":"2024-09-16T01:10:47.438556+0000","last_fullsized":"2024-09-16T01:10:47.438556+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271474+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:10:47.271474+0000","last_peered":"2024-09-16T01:10:47.271474+0000","last_clean":"2024-09-16T01:10:47.271474+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:10:47.271474+0000","last_undegraded":"2024-09-16T01:10:47.271474+0000","last_fullsized":"2024-09-16T01:10:47.271474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525822+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:10:47.525822+0000","last_peered":"2024-09-16T01:10:47.525822+0000","last_clean":"2024-09-16T01:10:47.525822+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:10:47.525822+0000","last_undegraded":"2024-09-16T01:10:47.525822+0000","last_fullsized":"2024-09-16T01:10:47.525822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201931+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:10:47.201931+0000","last_peered":"2024-09-16T01:10:47.201931+0000","last_clean":"2024-09-16T01:10:47.201931+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:10:47.201931+0000","last_undegraded":"2024-09-16T01:10:47.201931+0000","last_fullsized":"2024-09-16T01:10:47.201931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287352+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:10:47.287352+0000","last_peered":"2024-09-16T01:10:47.287352+0000","last_clean":"2024-09-16T01:10:47.287352+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:10:47.287352+0000","last_undegraded":"2024-09-16T01:10:47.287352+0000","last_fullsized":"2024-09-16T01:10:47.287352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525997+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:10:47.525997+0000","last_peered":"2024-09-16T01:10:47.525997+0000","last_clean":"2024-09-16T01:10:47.525997+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:10:47.525997+0000","last_undegraded":"2024-09-16T01:10:47.525997+0000","last_fullsized":"2024-09-16T01:10:47.525997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438123+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:10:47.438123+0000","last_peered":"2024-09-16T01:10:47.438123+0000","last_clean":"2024-09-16T01:10:47.438123+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:10:47.438123+0000","last_undegraded":"2024-09-16T01:10:47.438123+0000","last_fullsized":"2024-09-16T01:10:47.438123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525723+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:10:47.525723+0000","last_peered":"2024-09-16T01:10:47.525723+0000","last_clean":"2024-09-16T01:10:47.525723+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:10:47.525723+0000","last_undegraded":"2024-09-16T01:10:47.525723+0000","last_fullsized":"2024-09-16T01:10:47.525723+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525751+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:10:47.525751+0000","last_peered":"2024-09-16T01:10:47.525751+0000","last_clean":"2024-09-16T01:10:47.525751+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:10:47.525751+0000","last_undegraded":"2024-09-16T01:10:47.525751+0000","last_fullsized":"2024-09-16T01:10:47.525751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271378+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:10:47.271378+0000","last_peered":"2024-09-16T01:10:47.271378+0000","last_clean":"2024-09-16T01:10:47.271378+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:10:47.271378+0000","last_undegraded":"2024-09-16T01:10:47.271378+0000","last_fullsized":"2024-09-16T01:10:47.271378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198700+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:10:47.198700+0000","last_peered":"2024-09-16T01:10:47.198700+0000","last_clean":"2024-09-16T01:10:47.198700+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:10:47.198700+0000","last_undegraded":"2024-09-16T01:10:47.198700+0000","last_fullsized":"2024-09-16T01:10:47.198700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442364+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:10:47.442364+0000","last_peered":"2024-09-16T01:10:47.442364+0000","last_clean":"2024-09-16T01:10:47.442364+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:10:47.442364+0000","last_undegraded":"2024-09-16T01:10:47.442364+0000","last_fullsized":"2024-09-16T01:10:47.442364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266799+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:10:47.266799+0000","last_peered":"2024-09-16T01:10:47.266799+0000","last_clean":"2024-09-16T01:10:47.266799+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:10:47.266799+0000","last_undegraded":"2024-09-16T01:10:47.266799+0000","last_fullsized":"2024-09-16T01:10:47.266799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.439065+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:10:47.439065+0000","last_peered":"2024-09-16T01:10:47.439065+0000","last_clean":"2024-09-16T01:10:47.439065+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:10:47.439065+0000","last_undegraded":"2024-09-16T01:10:47.439065+0000","last_fullsized":"2024-09-16T01:10:47.439065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197771+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:10:47.197771+0000","last_peered":"2024-09-16T01:10:47.197771+0000","last_clean":"2024-09-16T01:10:47.197771+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:10:47.197771+0000","last_undegraded":"2024-09-16T01:10:47.197771+0000","last_fullsized":"2024-09-16T01:10:47.197771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442000+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:10:47.442000+0000","last_peered":"2024-09-16T01:10:47.442000+0000","last_clean":"2024-09-16T01:10:47.442000+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:10:47.442000+0000","last_undegraded":"2024-09-16T01:10:47.442000+0000","last_fullsized":"2024-09-16T01:10:47.442000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266403+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:10:47.266403+0000","last_peered":"2024-09-16T01:10:47.266403+0000","last_clean":"2024-09-16T01:10:47.266403+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:10:47.266403+0000","last_undegraded":"2024-09-16T01:10:47.266403+0000","last_fullsized":"2024-09-16T01:10:47.266403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289667+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:10:47.289667+0000","last_peered":"2024-09-16T01:10:47.289667+0000","last_clean":"2024-09-16T01:10:47.289667+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:10:47.289667+0000","last_undegraded":"2024-09-16T01:10:47.289667+0000","last_fullsized":"2024-09-16T01:10:47.289667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198621+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:10:47.198621+0000","last_peered":"2024-09-16T01:10:47.198621+0000","last_clean":"2024-09-16T01:10:47.198621+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:10:47.198621+0000","last_undegraded":"2024-09-16T01:10:47.198621+0000","last_fullsized":"2024-09-16T01:10:47.198621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525772+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:10:47.525772+0000","last_peered":"2024-09-16T01:10:47.525772+0000","last_clean":"2024-09-16T01:10:47.525772+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:10:47.525772+0000","last_undegraded":"2024-09-16T01:10:47.525772+0000","last_fullsized":"2024-09-16T01:10:47.525772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526118+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:10:47.526118+0000","last_peered":"2024-09-16T01:10:47.526118+0000","last_clean":"2024-09-16T01:10:47.526118+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:10:47.526118+0000","last_undegraded":"2024-09-16T01:10:47.526118+0000","last_fullsized":"2024-09-16T01:10:47.526118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525943+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:10:47.525943+0000","last_peered":"2024-09-16T01:10:47.525943+0000","last_clean":"2024-09-16T01:10:47.525943+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:10:47.525943+0000","last_undegraded":"2024-09-16T01:10:47.525943+0000","last_fullsized":"2024-09-16T01:10:47.525943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799177,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1017,"num_read_kb":722,"num_write":430,"num_write_kb":2364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":634880,"data_stored":2197200,"data_compressed":94141,"data_compressed_allocated":122880,"data_compressed_original":1916928,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107904,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481835,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1040384,"data_stored":537345,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710767,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712764,"statfs":{"total":95995035648,"available":95961870336,"internally_reserved":0,"allocated":1232896,"data_stored":1523624,"data_compressed":83561,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874292,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":934859,"data_compressed":51680,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103224,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":930683,"data_compressed":51537,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1101824,"data_stored":1134613,"data_compressed":68359,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528385,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1032192,"data_stored":532723,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1155072,"data_stored":1457591,"data_compressed":83561,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":539113,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1044480,"data_stored":532917,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":603808,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1105920,"data_stored":601425,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1056768,"data_stored":534111,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":73695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330494,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":403143,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:10:54.954 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-16T01:10:54.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-16T01:10:55.209 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:55 smithi078 ceph-mon[30937]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 184 KiB/s wr, 82 op/s 2024-09-16T01:10:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:55 smithi078 ceph-mon[30937]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:55 smithi148 ceph-mon[35030]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 184 KiB/s wr, 82 op/s 2024-09-16T01:10:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:55 smithi148 ceph-mon[35030]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:55 smithi193 ceph-mon[35701]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 184 KiB/s wr, 82 op/s 2024-09-16T01:10:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:55 smithi193 ceph-mon[35701]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:55.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 -- 172.21.15.78:0/3061966545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f8102d50 msgr2=0x7f18f8103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/3061966545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f8102d50 0x7f18f8103130 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009ca0 tx=0x7f18e802a950 comp rx=0 tx=0).stop 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 -- 172.21.15.78:0/3061966545 shutdown_connections 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/3061966545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f810c180 0x7f18f810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/3061966545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8103670 0x7f18f810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/3061966545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f8102d50 0x7f18f8103130 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.678+0000 7f18ff7f9640 1 -- 172.21.15.78:0/3061966545 >> 172.21.15.78:0/3061966545 conn(0x7f18f80fc7d0 msgr2=0x7f18f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- 172.21.15.78:0/3061966545 shutdown_connections 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- 172.21.15.78:0/3061966545 wait complete. 2024-09-16T01:10:55.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 Processor -- start 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- start start 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8102d50 0x7f18f8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f8103670 0x7f18f81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18f8111fc0 con 0x7f18f810c180 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18f8111e40 con 0x7f18f8102d50 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.679+0000 7f18ff7f9640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18f8112140 con 0x7f18f8103670 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47784/0 (socket says 172.21.15.78:47784) 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 -- 172.21.15.78:0/1113087470 learned_addr learned my addr 172.21.15.78:0/1113087470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fd595640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8102d50 0x7f18f8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fcd94640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f8103670 0x7f18f81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 -- 172.21.15.78:0/1113087470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f8103670 msgr2=0x7f18f81989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f8103670 0x7f18f81989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 -- 172.21.15.78:0/1113087470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8102d50 msgr2=0x7f18f8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8102d50 0x7f18f8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 -- 172.21.15.78:0/1113087470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f819d410 con 0x7f18f810c180 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18fdd96640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009fd0 tx=0x7f18f000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:55.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0011830 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18f0016070 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f819d700 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f000f2e0 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.680+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f81a4fe0 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.681+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f8103130 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.682+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 24) ==== 138582+0+0 (secure 0 0 0) 0x7f18f00119d0 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.683+0000 7f18f67fc640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 0x7f18e00a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.683+0000 7f18fd595640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 0x7f18e00a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.683+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f18f00cff80 con 0x7f18f810c180 2024-09-16T01:10:55.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.684+0000 7f18fd595640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 0x7f18e00a5990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18e8004ad0 tx=0x7f18e8004a60 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:55.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.686+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18f009c490 con 0x7f18f810c180 2024-09-16T01:10:55.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.850+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18f8062630 con 0x7f18e00a34d0 2024-09-16T01:10:55.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.858+0000 7f18f67fc640 1 -- 172.21.15.78:0/1113087470 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339710 (secure 0 0 0) 0x7f18f8062630 con 0x7f18e00a34d0 2024-09-16T01:10:55.863 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-16T01:10:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 msgr2=0x7f18e00a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 0x7f18e00a5990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18e8004ad0 tx=0x7f18e8004a60 comp rx=0 tx=0).stop 2024-09-16T01:10:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 msgr2=0x7f18f819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009fd0 tx=0x7f18f000b8e0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 shutdown_connections 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18e00a34d0 0x7f18e00a5990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18f810c180 0x7f18f819cd30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18f8103670 0x7f18f81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 --2- 172.21.15.78:0/1113087470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18f8102d50 0x7f18f8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 >> 172.21.15.78:0/1113087470 conn(0x7f18f80fc7d0 msgr2=0x7f18f80fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 shutdown_connections 2024-09-16T01:10:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:55.863+0000 7f18ff7f9640 1 -- 172.21.15.78:0/1113087470 wait complete. 2024-09-16T01:10:56.577 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-09-16T01:10:55.216343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389779,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1259,"num_read_kb":1057,"num_write":635,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":967,"ondisk_log_size":967,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382944,"kb_used_data":12864,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558880,"statfs":{"total":1151940427776,"available":1151548293120,"internally_reserved":0,"allocated":13172736,"data_stored":10587959,"data_compressed":600069,"data_compressed_allocated":811008,"data_compressed_original":4952064,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":200331,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":387,"num_read_kb":465,"num_write":154,"num_write_kb":922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.997904"},"pg_stats":[{"pgid":"2.1f","version":"89'9","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287576+0000","last_change":"2024-09-16T01:08:26.712513+0000","last_active":"2024-09-16T01:10:47.287576+0000","last_peered":"2024-09-16T01:10:47.287576+0000","last_clean":"2024-09-16T01:10:47.287576+0000","last_became_active":"2024-09-16T01:08:26.712244+0000","last_became_peered":"2024-09-16T01:08:26.712244+0000","last_unstale":"2024-09-16T01:10:47.287576+0000","last_undegraded":"2024-09-16T01:10:47.287576+0000","last_fullsized":"2024-09-16T01:10:47.287576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:21.043696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198446+0000","last_change":"2024-09-16T01:08:28.726697+0000","last_active":"2024-09-16T01:10:47.198446+0000","last_peered":"2024-09-16T01:10:47.198446+0000","last_clean":"2024-09-16T01:10:47.198446+0000","last_became_active":"2024-09-16T01:08:28.726404+0000","last_became_peered":"2024-09-16T01:08:28.726404+0000","last_unstale":"2024-09-16T01:10:47.198446+0000","last_undegraded":"2024-09-16T01:10:47.198446+0000","last_fullsized":"2024-09-16T01:10:47.198446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:41.555422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201248+0000","last_change":"2024-09-16T01:08:35.772057+0000","last_active":"2024-09-16T01:10:47.201248+0000","last_peered":"2024-09-16T01:10:47.201248+0000","last_clean":"2024-09-16T01:10:47.201248+0000","last_became_active":"2024-09-16T01:08:35.771476+0000","last_became_peered":"2024-09-16T01:08:35.771476+0000","last_unstale":"2024-09-16T01:10:47.201248+0000","last_undegraded":"2024-09-16T01:10:47.201248+0000","last_fullsized":"2024-09-16T01:10:47.201248+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:33.167751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198687+0000","last_change":"2024-09-16T01:08:26.955297+0000","last_active":"2024-09-16T01:10:47.198687+0000","last_peered":"2024-09-16T01:10:47.198687+0000","last_clean":"2024-09-16T01:10:47.198687+0000","last_became_active":"2024-09-16T01:08:26.954968+0000","last_became_peered":"2024-09-16T01:08:26.954968+0000","last_unstale":"2024-09-16T01:10:47.198687+0000","last_undegraded":"2024-09-16T01:10:47.198687+0000","last_fullsized":"2024-09-16T01:10:47.198687+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:05.851862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526283+0000","last_change":"2024-09-16T01:08:28.719935+0000","last_active":"2024-09-16T01:10:47.526283+0000","last_peered":"2024-09-16T01:10:47.526283+0000","last_clean":"2024-09-16T01:10:47.526283+0000","last_became_active":"2024-09-16T01:08:28.719626+0000","last_became_peered":"2024-09-16T01:08:28.719626+0000","last_unstale":"2024-09-16T01:10:47.526283+0000","last_undegraded":"2024-09-16T01:10:47.526283+0000","last_fullsized":"2024-09-16T01:10:47.526283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:06.768261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198235+0000","last_change":"2024-09-16T01:08:35.768506+0000","last_active":"2024-09-16T01:10:47.198235+0000","last_peered":"2024-09-16T01:10:47.198235+0000","last_clean":"2024-09-16T01:10:47.198235+0000","last_became_active":"2024-09-16T01:08:35.768070+0000","last_became_peered":"2024-09-16T01:08:35.768070+0000","last_unstale":"2024-09-16T01:10:47.198235+0000","last_undegraded":"2024-09-16T01:10:47.198235+0000","last_fullsized":"2024-09-16T01:10:47.198235+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:10.377733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525633+0000","last_change":"2024-09-16T01:10:40.108958+0000","last_active":"2024-09-16T01:10:47.525633+0000","last_peered":"2024-09-16T01:10:47.525633+0000","last_clean":"2024-09-16T01:10:47.525633+0000","last_became_active":"2024-09-16T01:10:40.099865+0000","last_became_peered":"2024-09-16T01:10:40.099865+0000","last_unstale":"2024-09-16T01:10:47.525633+0000","last_undegraded":"2024-09-16T01:10:47.525633+0000","last_fullsized":"2024-09-16T01:10:47.525633+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:48.312221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438311+0000","last_change":"2024-09-16T01:08:28.713450+0000","last_active":"2024-09-16T01:10:47.438311+0000","last_peered":"2024-09-16T01:10:47.438311+0000","last_clean":"2024-09-16T01:10:47.438311+0000","last_became_active":"2024-09-16T01:08:28.713117+0000","last_became_peered":"2024-09-16T01:08:28.713117+0000","last_unstale":"2024-09-16T01:10:47.438311+0000","last_undegraded":"2024-09-16T01:10:47.438311+0000","last_fullsized":"2024-09-16T01:10:47.438311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:09.751218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198398+0000","last_change":"2024-09-16T01:08:35.769640+0000","last_active":"2024-09-16T01:10:47.198398+0000","last_peered":"2024-09-16T01:10:47.198398+0000","last_clean":"2024-09-16T01:10:47.198398+0000","last_became_active":"2024-09-16T01:08:35.769005+0000","last_became_peered":"2024-09-16T01:08:35.769005+0000","last_unstale":"2024-09-16T01:10:47.198398+0000","last_undegraded":"2024-09-16T01:10:47.198398+0000","last_fullsized":"2024-09-16T01:10:47.198398+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:52.280809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"83'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290079+0000","last_change":"2024-09-16T01:08:26.706492+0000","last_active":"2024-09-16T01:10:47.290079+0000","last_peered":"2024-09-16T01:10:47.290079+0000","last_clean":"2024-09-16T01:10:47.290079+0000","last_became_active":"2024-09-16T01:08:26.705561+0000","last_became_peered":"2024-09-16T01:08:26.705561+0000","last_unstale":"2024-09-16T01:10:47.290079+0000","last_undegraded":"2024-09-16T01:10:47.290079+0000","last_fullsized":"2024-09-16T01:10:47.290079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:17.224246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269004+0000","last_change":"2024-09-16T01:08:28.728034+0000","last_active":"2024-09-16T01:10:47.269004+0000","last_peered":"2024-09-16T01:10:47.269004+0000","last_clean":"2024-09-16T01:10:47.269004+0000","last_became_active":"2024-09-16T01:08:28.727693+0000","last_became_peered":"2024-09-16T01:08:28.727693+0000","last_unstale":"2024-09-16T01:10:47.269004+0000","last_undegraded":"2024-09-16T01:10:47.269004+0000","last_fullsized":"2024-09-16T01:10:47.269004+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:50.398483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198272+0000","last_change":"2024-09-16T01:08:35.770958+0000","last_active":"2024-09-16T01:10:47.198272+0000","last_peered":"2024-09-16T01:10:47.198272+0000","last_clean":"2024-09-16T01:10:47.198272+0000","last_became_active":"2024-09-16T01:08:35.770437+0000","last_became_peered":"2024-09-16T01:08:35.770437+0000","last_unstale":"2024-09-16T01:10:47.198272+0000","last_undegraded":"2024-09-16T01:10:47.198272+0000","last_fullsized":"2024-09-16T01:10:47.198272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:24.537744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442368+0000","last_change":"2024-09-16T01:08:28.723291+0000","last_active":"2024-09-16T01:10:47.442368+0000","last_peered":"2024-09-16T01:10:47.442368+0000","last_clean":"2024-09-16T01:10:47.442368+0000","last_became_active":"2024-09-16T01:08:28.722921+0000","last_became_peered":"2024-09-16T01:08:28.722921+0000","last_unstale":"2024-09-16T01:10:47.442368+0000","last_undegraded":"2024-09-16T01:10:47.442368+0000","last_fullsized":"2024-09-16T01:10:47.442368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:09.972773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289720+0000","last_change":"2024-09-16T01:08:26.713503+0000","last_active":"2024-09-16T01:10:47.289720+0000","last_peered":"2024-09-16T01:10:47.289720+0000","last_clean":"2024-09-16T01:10:47.289720+0000","last_became_active":"2024-09-16T01:08:26.713129+0000","last_became_peered":"2024-09-16T01:08:26.713129+0000","last_unstale":"2024-09-16T01:10:47.289720+0000","last_undegraded":"2024-09-16T01:10:47.289720+0000","last_fullsized":"2024-09-16T01:10:47.289720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:13.343103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198767+0000","last_change":"2024-09-16T01:08:36.346182+0000","last_active":"2024-09-16T01:10:47.198767+0000","last_peered":"2024-09-16T01:10:47.198767+0000","last_clean":"2024-09-16T01:10:47.198767+0000","last_became_active":"2024-09-16T01:08:36.345341+0000","last_became_peered":"2024-09-16T01:08:36.345341+0000","last_unstale":"2024-09-16T01:10:47.198767+0000","last_undegraded":"2024-09-16T01:10:47.198767+0000","last_fullsized":"2024-09-16T01:10:47.198767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:29.134751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198351+0000","last_change":"2024-09-16T01:08:28.730349+0000","last_active":"2024-09-16T01:10:47.198351+0000","last_peered":"2024-09-16T01:10:47.198351+0000","last_clean":"2024-09-16T01:10:47.198351+0000","last_became_active":"2024-09-16T01:08:28.730045+0000","last_became_peered":"2024-09-16T01:08:28.730045+0000","last_unstale":"2024-09-16T01:10:47.198351+0000","last_undegraded":"2024-09-16T01:10:47.198351+0000","last_fullsized":"2024-09-16T01:10:47.198351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:58.618171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201616+0000","last_change":"2024-09-16T01:08:26.710434+0000","last_active":"2024-09-16T01:10:47.201616+0000","last_peered":"2024-09-16T01:10:47.201616+0000","last_clean":"2024-09-16T01:10:47.201616+0000","last_became_active":"2024-09-16T01:08:26.709193+0000","last_became_peered":"2024-09-16T01:08:26.709193+0000","last_unstale":"2024-09-16T01:10:47.201616+0000","last_undegraded":"2024-09-16T01:10:47.201616+0000","last_fullsized":"2024-09-16T01:10:47.201616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:47.830103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198574+0000","last_change":"2024-09-16T01:08:35.769319+0000","last_active":"2024-09-16T01:10:47.198574+0000","last_peered":"2024-09-16T01:10:47.198574+0000","last_clean":"2024-09-16T01:10:47.198574+0000","last_became_active":"2024-09-16T01:08:35.768862+0000","last_became_peered":"2024-09-16T01:08:35.768862+0000","last_unstale":"2024-09-16T01:10:47.198574+0000","last_undegraded":"2024-09-16T01:10:47.198574+0000","last_fullsized":"2024-09-16T01:10:47.198574+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:52.849827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525563+0000","last_change":"2024-09-16T01:08:28.719408+0000","last_active":"2024-09-16T01:10:47.525563+0000","last_peered":"2024-09-16T01:10:47.525563+0000","last_clean":"2024-09-16T01:10:47.525563+0000","last_became_active":"2024-09-16T01:08:28.719029+0000","last_became_peered":"2024-09-16T01:08:28.719029+0000","last_unstale":"2024-09-16T01:10:47.525563+0000","last_undegraded":"2024-09-16T01:10:47.525563+0000","last_fullsized":"2024-09-16T01:10:47.525563+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:05.660599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198104+0000","last_change":"2024-09-16T01:08:26.710214+0000","last_active":"2024-09-16T01:10:47.198104+0000","last_peered":"2024-09-16T01:10:47.198104+0000","last_clean":"2024-09-16T01:10:47.198104+0000","last_became_active":"2024-09-16T01:08:26.709873+0000","last_became_peered":"2024-09-16T01:08:26.709873+0000","last_unstale":"2024-09-16T01:10:47.198104+0000","last_undegraded":"2024-09-16T01:10:47.198104+0000","last_fullsized":"2024-09-16T01:10:47.198104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.538528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287247+0000","last_change":"2024-09-16T01:08:35.771126+0000","last_active":"2024-09-16T01:10:47.287247+0000","last_peered":"2024-09-16T01:10:47.287247+0000","last_clean":"2024-09-16T01:10:47.287247+0000","last_became_active":"2024-09-16T01:08:35.770270+0000","last_became_peered":"2024-09-16T01:08:35.770270+0000","last_unstale":"2024-09-16T01:10:47.287247+0000","last_undegraded":"2024-09-16T01:10:47.287247+0000","last_fullsized":"2024-09-16T01:10:47.287247+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:09.780571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289566+0000","last_change":"2024-09-16T01:08:28.717153+0000","last_active":"2024-09-16T01:10:47.289566+0000","last_peered":"2024-09-16T01:10:47.289566+0000","last_clean":"2024-09-16T01:10:47.289566+0000","last_became_active":"2024-09-16T01:08:28.716803+0000","last_became_peered":"2024-09-16T01:08:28.716803+0000","last_unstale":"2024-09-16T01:10:47.289566+0000","last_undegraded":"2024-09-16T01:10:47.289566+0000","last_fullsized":"2024-09-16T01:10:47.289566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:40.588176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438269+0000","last_change":"2024-09-16T01:08:26.955611+0000","last_active":"2024-09-16T01:10:47.438269+0000","last_peered":"2024-09-16T01:10:47.438269+0000","last_clean":"2024-09-16T01:10:47.438269+0000","last_became_active":"2024-09-16T01:08:26.955294+0000","last_became_peered":"2024-09-16T01:08:26.955294+0000","last_unstale":"2024-09-16T01:10:47.438269+0000","last_undegraded":"2024-09-16T01:10:47.438269+0000","last_fullsized":"2024-09-16T01:10:47.438269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:22.130935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198841+0000","last_change":"2024-09-16T01:08:36.345965+0000","last_active":"2024-09-16T01:10:47.198841+0000","last_peered":"2024-09-16T01:10:47.198841+0000","last_clean":"2024-09-16T01:10:47.198841+0000","last_became_active":"2024-09-16T01:08:36.345473+0000","last_became_peered":"2024-09-16T01:08:36.345473+0000","last_unstale":"2024-09-16T01:10:47.198841+0000","last_undegraded":"2024-09-16T01:10:47.198841+0000","last_fullsized":"2024-09-16T01:10:47.198841+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:16.409617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526265+0000","last_change":"2024-09-16T01:08:28.722514+0000","last_active":"2024-09-16T01:10:47.526265+0000","last_peered":"2024-09-16T01:10:47.526265+0000","last_clean":"2024-09-16T01:10:47.526265+0000","last_became_active":"2024-09-16T01:08:28.722208+0000","last_became_peered":"2024-09-16T01:08:28.722208+0000","last_unstale":"2024-09-16T01:10:47.526265+0000","last_undegraded":"2024-09-16T01:10:47.526265+0000","last_fullsized":"2024-09-16T01:10:47.526265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:20.355146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198806+0000","last_change":"2024-09-16T01:08:26.709446+0000","last_active":"2024-09-16T01:10:47.198806+0000","last_peered":"2024-09-16T01:10:47.198806+0000","last_clean":"2024-09-16T01:10:47.198806+0000","last_became_active":"2024-09-16T01:08:26.708441+0000","last_became_peered":"2024-09-16T01:08:26.708441+0000","last_unstale":"2024-09-16T01:10:47.198806+0000","last_undegraded":"2024-09-16T01:10:47.198806+0000","last_fullsized":"2024-09-16T01:10:47.198806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:11.590312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201469+0000","last_change":"2024-09-16T01:08:35.766407+0000","last_active":"2024-09-16T01:10:47.201469+0000","last_peered":"2024-09-16T01:10:47.201469+0000","last_clean":"2024-09-16T01:10:47.201469+0000","last_became_active":"2024-09-16T01:08:35.765947+0000","last_became_peered":"2024-09-16T01:08:35.765947+0000","last_unstale":"2024-09-16T01:10:47.201469+0000","last_undegraded":"2024-09-16T01:10:47.201469+0000","last_fullsized":"2024-09-16T01:10:47.201469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:45.053104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198893+0000","last_change":"2024-09-16T01:08:28.724717+0000","last_active":"2024-09-16T01:10:47.198893+0000","last_peered":"2024-09-16T01:10:47.198893+0000","last_clean":"2024-09-16T01:10:47.198893+0000","last_became_active":"2024-09-16T01:08:28.724378+0000","last_became_peered":"2024-09-16T01:08:28.724378+0000","last_unstale":"2024-09-16T01:10:47.198893+0000","last_undegraded":"2024-09-16T01:10:47.198893+0000","last_fullsized":"2024-09-16T01:10:47.198893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:16.790146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'104","reported_seq":237,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:50.800752+0000","last_change":"2024-09-16T01:08:26.705686+0000","last_active":"2024-09-16T01:10:50.800752+0000","last_peered":"2024-09-16T01:10:50.800752+0000","last_clean":"2024-09-16T01:10:50.800752+0000","last_became_active":"2024-09-16T01:08:26.705362+0000","last_became_peered":"2024-09-16T01:08:26.705362+0000","last_unstale":"2024-09-16T01:10:50.800752+0000","last_undegraded":"2024-09-16T01:10:50.800752+0000","last_fullsized":"2024-09-16T01:10:50.800752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:22.887640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":191,"num_write_kb":1289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197770+0000","last_change":"2024-09-16T01:08:36.345457+0000","last_active":"2024-09-16T01:10:47.197770+0000","last_peered":"2024-09-16T01:10:47.197770+0000","last_clean":"2024-09-16T01:10:47.197770+0000","last_became_active":"2024-09-16T01:08:36.345034+0000","last_became_peered":"2024-09-16T01:08:36.345034+0000","last_unstale":"2024-09-16T01:10:47.197770+0000","last_undegraded":"2024-09-16T01:10:47.197770+0000","last_fullsized":"2024-09-16T01:10:47.197770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:38:20.195641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271664+0000","last_change":"2024-09-16T01:08:28.723805+0000","last_active":"2024-09-16T01:10:47.271664+0000","last_peered":"2024-09-16T01:10:47.271664+0000","last_clean":"2024-09-16T01:10:47.271664+0000","last_became_active":"2024-09-16T01:08:28.723517+0000","last_became_peered":"2024-09-16T01:08:28.723517+0000","last_unstale":"2024-09-16T01:10:47.271664+0000","last_undegraded":"2024-09-16T01:10:47.271664+0000","last_fullsized":"2024-09-16T01:10:47.271664+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:57:36.636961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.290000+0000","last_change":"2024-09-16T01:08:26.955930+0000","last_active":"2024-09-16T01:10:47.290000+0000","last_peered":"2024-09-16T01:10:47.290000+0000","last_clean":"2024-09-16T01:10:47.290000+0000","last_became_active":"2024-09-16T01:08:26.955513+0000","last_became_peered":"2024-09-16T01:08:26.955513+0000","last_unstale":"2024-09-16T01:10:47.290000+0000","last_undegraded":"2024-09-16T01:10:47.290000+0000","last_fullsized":"2024-09-16T01:10:47.290000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:38.769690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525655+0000","last_change":"2024-09-16T01:08:35.773921+0000","last_active":"2024-09-16T01:10:47.525655+0000","last_peered":"2024-09-16T01:10:47.525655+0000","last_clean":"2024-09-16T01:10:47.525655+0000","last_became_active":"2024-09-16T01:08:35.773268+0000","last_became_peered":"2024-09-16T01:08:35.773268+0000","last_unstale":"2024-09-16T01:10:47.525655+0000","last_undegraded":"2024-09-16T01:10:47.525655+0000","last_fullsized":"2024-09-16T01:10:47.525655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:15.559289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266701+0000","last_change":"2024-09-16T01:08:28.719050+0000","last_active":"2024-09-16T01:10:47.266701+0000","last_peered":"2024-09-16T01:10:47.266701+0000","last_clean":"2024-09-16T01:10:47.266701+0000","last_became_active":"2024-09-16T01:08:28.715434+0000","last_became_peered":"2024-09-16T01:08:28.715434+0000","last_unstale":"2024-09-16T01:10:47.266701+0000","last_undegraded":"2024-09-16T01:10:47.266701+0000","last_fullsized":"2024-09-16T01:10:47.266701+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:44.869802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286991+0000","last_change":"2024-09-16T01:08:26.955708+0000","last_active":"2024-09-16T01:10:47.286991+0000","last_peered":"2024-09-16T01:10:47.286991+0000","last_clean":"2024-09-16T01:10:47.286991+0000","last_became_active":"2024-09-16T01:08:26.955362+0000","last_became_peered":"2024-09-16T01:08:26.955362+0000","last_unstale":"2024-09-16T01:10:47.286991+0000","last_undegraded":"2024-09-16T01:10:47.286991+0000","last_fullsized":"2024-09-16T01:10:47.286991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:02.651685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289584+0000","last_change":"2024-09-16T01:09:39.117797+0000","last_active":"2024-09-16T01:10:47.289584+0000","last_peered":"2024-09-16T01:10:47.289584+0000","last_clean":"2024-09-16T01:10:47.289584+0000","last_became_active":"2024-09-16T01:09:39.117345+0000","last_became_peered":"2024-09-16T01:09:39.117345+0000","last_unstale":"2024-09-16T01:10:47.289584+0000","last_undegraded":"2024-09-16T01:10:47.289584+0000","last_fullsized":"2024-09-16T01:10:47.289584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:21.410065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266431+0000","last_change":"2024-09-16T01:09:39.148653+0000","last_active":"2024-09-16T01:10:47.266431+0000","last_peered":"2024-09-16T01:10:47.266431+0000","last_clean":"2024-09-16T01:10:47.266431+0000","last_became_active":"2024-09-16T01:09:39.148312+0000","last_became_peered":"2024-09-16T01:09:39.148312+0000","last_unstale":"2024-09-16T01:10:47.266431+0000","last_undegraded":"2024-09-16T01:10:47.266431+0000","last_fullsized":"2024-09-16T01:10:47.266431+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:17.031741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442537+0000","last_change":"2024-09-16T01:08:26.714135+0000","last_active":"2024-09-16T01:10:47.442537+0000","last_peered":"2024-09-16T01:10:47.442537+0000","last_clean":"2024-09-16T01:10:47.442537+0000","last_became_active":"2024-09-16T01:08:26.713792+0000","last_became_peered":"2024-09-16T01:08:26.713792+0000","last_unstale":"2024-09-16T01:10:47.442537+0000","last_undegraded":"2024-09-16T01:10:47.442537+0000","last_fullsized":"2024-09-16T01:10:47.442537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:01.422951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271403+0000","last_change":"2024-09-16T01:08:35.773113+0000","last_active":"2024-09-16T01:10:47.271403+0000","last_peered":"2024-09-16T01:10:47.271403+0000","last_clean":"2024-09-16T01:10:47.271403+0000","last_became_active":"2024-09-16T01:08:35.772720+0000","last_became_peered":"2024-09-16T01:08:35.772720+0000","last_unstale":"2024-09-16T01:10:47.271403+0000","last_undegraded":"2024-09-16T01:10:47.271403+0000","last_fullsized":"2024-09-16T01:10:47.271403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:22.489922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268883+0000","last_change":"2024-09-16T01:08:28.722871+0000","last_active":"2024-09-16T01:10:47.268883+0000","last_peered":"2024-09-16T01:10:47.268883+0000","last_clean":"2024-09-16T01:10:47.268883+0000","last_became_active":"2024-09-16T01:08:28.722503+0000","last_became_peered":"2024-09-16T01:08:28.722503+0000","last_unstale":"2024-09-16T01:10:47.268883+0000","last_undegraded":"2024-09-16T01:10:47.268883+0000","last_fullsized":"2024-09-16T01:10:47.268883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:41.049805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":153,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:51.276825+0000","last_change":"2024-09-16T01:08:26.955280+0000","last_active":"2024-09-16T01:10:51.276825+0000","last_peered":"2024-09-16T01:10:51.276825+0000","last_clean":"2024-09-16T01:10:51.276825+0000","last_became_active":"2024-09-16T01:08:26.954937+0000","last_became_peered":"2024-09-16T01:08:26.954937+0000","last_unstale":"2024-09-16T01:10:51.276825+0000","last_undegraded":"2024-09-16T01:10:51.276825+0000","last_fullsized":"2024-09-16T01:10:51.276825+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:42.694725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272128+0000","last_change":"2024-09-16T01:08:36.345551+0000","last_active":"2024-09-16T01:10:47.272128+0000","last_peered":"2024-09-16T01:10:47.272128+0000","last_clean":"2024-09-16T01:10:47.272128+0000","last_became_active":"2024-09-16T01:08:36.345159+0000","last_became_peered":"2024-09-16T01:08:36.345159+0000","last_unstale":"2024-09-16T01:10:47.272128+0000","last_undegraded":"2024-09-16T01:10:47.272128+0000","last_fullsized":"2024-09-16T01:10:47.272128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:40.011455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271097+0000","last_change":"2024-09-16T01:08:28.721407+0000","last_active":"2024-09-16T01:10:47.271097+0000","last_peered":"2024-09-16T01:10:47.271097+0000","last_clean":"2024-09-16T01:10:47.271097+0000","last_became_active":"2024-09-16T01:08:28.721121+0000","last_became_peered":"2024-09-16T01:08:28.721121+0000","last_unstale":"2024-09-16T01:10:47.271097+0000","last_undegraded":"2024-09-16T01:10:47.271097+0000","last_fullsized":"2024-09-16T01:10:47.271097+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:31:56.865404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441989+0000","last_change":"2024-09-16T01:08:26.711246+0000","last_active":"2024-09-16T01:10:47.441989+0000","last_peered":"2024-09-16T01:10:47.441989+0000","last_clean":"2024-09-16T01:10:47.441989+0000","last_became_active":"2024-09-16T01:08:26.710900+0000","last_became_peered":"2024-09-16T01:08:26.710900+0000","last_unstale":"2024-09-16T01:10:47.441989+0000","last_undegraded":"2024-09-16T01:10:47.441989+0000","last_fullsized":"2024-09-16T01:10:47.441989+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:32.144831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269086+0000","last_change":"2024-09-16T01:09:39.167998+0000","last_active":"2024-09-16T01:10:47.269086+0000","last_peered":"2024-09-16T01:10:47.269086+0000","last_clean":"2024-09-16T01:10:47.269086+0000","last_became_active":"2024-09-16T01:09:39.167533+0000","last_became_peered":"2024-09-16T01:09:39.167533+0000","last_unstale":"2024-09-16T01:10:47.269086+0000","last_undegraded":"2024-09-16T01:10:47.269086+0000","last_fullsized":"2024-09-16T01:10:47.269086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:32:01.974490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198237+0000","last_change":"2024-09-16T01:08:28.728486+0000","last_active":"2024-09-16T01:10:47.198237+0000","last_peered":"2024-09-16T01:10:47.198237+0000","last_clean":"2024-09-16T01:10:47.198237+0000","last_became_active":"2024-09-16T01:08:28.728187+0000","last_became_peered":"2024-09-16T01:08:28.728187+0000","last_unstale":"2024-09-16T01:10:47.198237+0000","last_undegraded":"2024-09-16T01:10:47.198237+0000","last_fullsized":"2024-09-16T01:10:47.198237+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:39.445853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198632+0000","last_change":"2024-09-16T01:08:26.709189+0000","last_active":"2024-09-16T01:10:47.198632+0000","last_peered":"2024-09-16T01:10:47.198632+0000","last_clean":"2024-09-16T01:10:47.198632+0000","last_became_active":"2024-09-16T01:08:26.708745+0000","last_became_peered":"2024-09-16T01:08:26.708745+0000","last_unstale":"2024-09-16T01:10:47.198632+0000","last_undegraded":"2024-09-16T01:10:47.198632+0000","last_fullsized":"2024-09-16T01:10:47.198632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:20.442190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":409,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:49.301392+0000","last_change":"2024-09-16T01:07:15.486000+0000","last_active":"2024-09-16T01:10:49.301392+0000","last_peered":"2024-09-16T01:10:49.301392+0000","last_clean":"2024-09-16T01:10:49.301392+0000","last_became_active":"2024-09-16T01:07:15.485544+0000","last_became_peered":"2024-09-16T01:07:15.485544+0000","last_unstale":"2024-09-16T01:10:49.301392+0000","last_undegraded":"2024-09-16T01:10:49.301392+0000","last_fullsized":"2024-09-16T01:10:49.301392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:03:52.280161+0000","last_clean_scrub_stamp":"2024-09-16T01:03:52.280161+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:55:54.860093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442226+0000","last_change":"2024-09-16T01:08:35.770940+0000","last_active":"2024-09-16T01:10:47.442226+0000","last_peered":"2024-09-16T01:10:47.442226+0000","last_clean":"2024-09-16T01:10:47.442226+0000","last_became_active":"2024-09-16T01:08:35.770516+0000","last_became_peered":"2024-09-16T01:08:35.770516+0000","last_unstale":"2024-09-16T01:10:47.442226+0000","last_undegraded":"2024-09-16T01:10:47.442226+0000","last_fullsized":"2024-09-16T01:10:47.442226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:10.972199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442414+0000","last_change":"2024-09-16T01:08:28.717525+0000","last_active":"2024-09-16T01:10:47.442414+0000","last_peered":"2024-09-16T01:10:47.442414+0000","last_clean":"2024-09-16T01:10:47.442414+0000","last_became_active":"2024-09-16T01:08:28.717156+0000","last_became_peered":"2024-09-16T01:08:28.717156+0000","last_unstale":"2024-09-16T01:10:47.442414+0000","last_undegraded":"2024-09-16T01:10:47.442414+0000","last_fullsized":"2024-09-16T01:10:47.442414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:41.916048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201739+0000","last_change":"2024-09-16T01:08:26.955637+0000","last_active":"2024-09-16T01:10:47.201739+0000","last_peered":"2024-09-16T01:10:47.201739+0000","last_clean":"2024-09-16T01:10:47.201739+0000","last_became_active":"2024-09-16T01:08:26.955258+0000","last_became_peered":"2024-09-16T01:08:26.955258+0000","last_unstale":"2024-09-16T01:10:47.201739+0000","last_undegraded":"2024-09-16T01:10:47.201739+0000","last_fullsized":"2024-09-16T01:10:47.201739+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:48.277010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438356+0000","last_change":"2024-09-16T01:08:35.773232+0000","last_active":"2024-09-16T01:10:47.438356+0000","last_peered":"2024-09-16T01:10:47.438356+0000","last_clean":"2024-09-16T01:10:47.438356+0000","last_became_active":"2024-09-16T01:08:35.772759+0000","last_became_peered":"2024-09-16T01:08:35.772759+0000","last_unstale":"2024-09-16T01:10:47.438356+0000","last_undegraded":"2024-09-16T01:10:47.438356+0000","last_fullsized":"2024-09-16T01:10:47.438356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.365973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442238+0000","last_change":"2024-09-16T01:08:28.728440+0000","last_active":"2024-09-16T01:10:47.442238+0000","last_peered":"2024-09-16T01:10:47.442238+0000","last_clean":"2024-09-16T01:10:47.442238+0000","last_became_active":"2024-09-16T01:08:28.727813+0000","last_became_peered":"2024-09-16T01:08:28.727813+0000","last_unstale":"2024-09-16T01:10:47.442238+0000","last_undegraded":"2024-09-16T01:10:47.442238+0000","last_fullsized":"2024-09-16T01:10:47.442238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:23.518347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":137,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:50.432697+0000","last_change":"2024-09-16T01:08:26.710624+0000","last_active":"2024-09-16T01:10:50.432697+0000","last_peered":"2024-09-16T01:10:50.432697+0000","last_clean":"2024-09-16T01:10:50.432697+0000","last_became_active":"2024-09-16T01:08:26.710189+0000","last_became_peered":"2024-09-16T01:08:26.710189+0000","last_unstale":"2024-09-16T01:10:50.432697+0000","last_undegraded":"2024-09-16T01:10:50.432697+0000","last_fullsized":"2024-09-16T01:10:50.432697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:39.302004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525865+0000","last_change":"2024-09-16T01:08:35.770930+0000","last_active":"2024-09-16T01:10:47.525865+0000","last_peered":"2024-09-16T01:10:47.525865+0000","last_clean":"2024-09-16T01:10:47.525865+0000","last_became_active":"2024-09-16T01:08:35.770552+0000","last_became_peered":"2024-09-16T01:08:35.770552+0000","last_unstale":"2024-09-16T01:10:47.525865+0000","last_undegraded":"2024-09-16T01:10:47.525865+0000","last_fullsized":"2024-09-16T01:10:47.525865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:40.258998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442274+0000","last_change":"2024-09-16T01:08:28.720026+0000","last_active":"2024-09-16T01:10:47.442274+0000","last_peered":"2024-09-16T01:10:47.442274+0000","last_clean":"2024-09-16T01:10:47.442274+0000","last_became_active":"2024-09-16T01:08:28.719700+0000","last_became_peered":"2024-09-16T01:08:28.719700+0000","last_unstale":"2024-09-16T01:10:47.442274+0000","last_undegraded":"2024-09-16T01:10:47.442274+0000","last_fullsized":"2024-09-16T01:10:47.442274+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:47.878131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.272022+0000","last_change":"2024-09-16T01:08:26.709064+0000","last_active":"2024-09-16T01:10:47.272022+0000","last_peered":"2024-09-16T01:10:47.272022+0000","last_clean":"2024-09-16T01:10:47.272022+0000","last_became_active":"2024-09-16T01:08:26.708734+0000","last_became_peered":"2024-09-16T01:08:26.708734+0000","last_unstale":"2024-09-16T01:10:47.272022+0000","last_undegraded":"2024-09-16T01:10:47.272022+0000","last_fullsized":"2024-09-16T01:10:47.272022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:23.776528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268804+0000","last_change":"2024-09-16T01:08:35.769109+0000","last_active":"2024-09-16T01:10:47.268804+0000","last_peered":"2024-09-16T01:10:47.268804+0000","last_clean":"2024-09-16T01:10:47.268804+0000","last_became_active":"2024-09-16T01:08:35.768507+0000","last_became_peered":"2024-09-16T01:08:35.768507+0000","last_unstale":"2024-09-16T01:10:47.268804+0000","last_undegraded":"2024-09-16T01:10:47.268804+0000","last_fullsized":"2024-09-16T01:10:47.268804+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:49.398637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525871+0000","last_change":"2024-09-16T01:09:39.119140+0000","last_active":"2024-09-16T01:10:47.525871+0000","last_peered":"2024-09-16T01:10:47.525871+0000","last_clean":"2024-09-16T01:10:47.525871+0000","last_became_active":"2024-09-16T01:09:39.118679+0000","last_became_peered":"2024-09-16T01:09:39.118679+0000","last_unstale":"2024-09-16T01:10:47.525871+0000","last_undegraded":"2024-09-16T01:10:47.525871+0000","last_fullsized":"2024-09-16T01:10:47.525871+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:14.355461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289958+0000","last_change":"2024-09-16T01:08:26.707531+0000","last_active":"2024-09-16T01:10:47.289958+0000","last_peered":"2024-09-16T01:10:47.289958+0000","last_clean":"2024-09-16T01:10:47.289958+0000","last_became_active":"2024-09-16T01:08:26.707213+0000","last_became_peered":"2024-09-16T01:08:26.707213+0000","last_unstale":"2024-09-16T01:10:47.289958+0000","last_undegraded":"2024-09-16T01:10:47.289958+0000","last_fullsized":"2024-09-16T01:10:47.289958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:04.863062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268774+0000","last_change":"2024-09-16T01:08:35.771967+0000","last_active":"2024-09-16T01:10:47.268774+0000","last_peered":"2024-09-16T01:10:47.268774+0000","last_clean":"2024-09-16T01:10:47.268774+0000","last_became_active":"2024-09-16T01:08:35.771513+0000","last_became_peered":"2024-09-16T01:08:35.771513+0000","last_unstale":"2024-09-16T01:10:47.268774+0000","last_undegraded":"2024-09-16T01:10:47.268774+0000","last_fullsized":"2024-09-16T01:10:47.268774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:59.567317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268924+0000","last_change":"2024-09-16T01:09:39.121178+0000","last_active":"2024-09-16T01:10:47.268924+0000","last_peered":"2024-09-16T01:10:47.268924+0000","last_clean":"2024-09-16T01:10:47.268924+0000","last_became_active":"2024-09-16T01:09:39.120781+0000","last_became_peered":"2024-09-16T01:09:39.120781+0000","last_unstale":"2024-09-16T01:10:47.268924+0000","last_undegraded":"2024-09-16T01:10:47.268924+0000","last_fullsized":"2024-09-16T01:10:47.268924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:43.670325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289670+0000","last_change":"2024-09-16T01:10:40.274011+0000","last_active":"2024-09-16T01:10:47.289670+0000","last_peered":"2024-09-16T01:10:47.289670+0000","last_clean":"2024-09-16T01:10:47.289670+0000","last_became_active":"2024-09-16T01:10:40.262422+0000","last_became_peered":"2024-09-16T01:10:40.262422+0000","last_unstale":"2024-09-16T01:10:47.289670+0000","last_undegraded":"2024-09-16T01:10:47.289670+0000","last_fullsized":"2024-09-16T01:10:47.289670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:32.563816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271887+0000","last_change":"2024-09-16T01:08:35.773782+0000","last_active":"2024-09-16T01:10:47.271887+0000","last_peered":"2024-09-16T01:10:47.271887+0000","last_clean":"2024-09-16T01:10:47.271887+0000","last_became_active":"2024-09-16T01:08:35.772143+0000","last_became_peered":"2024-09-16T01:08:35.772143+0000","last_unstale":"2024-09-16T01:10:47.271887+0000","last_undegraded":"2024-09-16T01:10:47.271887+0000","last_fullsized":"2024-09-16T01:10:47.271887+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:05.367514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.441971+0000","last_change":"2024-09-16T01:08:28.731917+0000","last_active":"2024-09-16T01:10:47.441971+0000","last_peered":"2024-09-16T01:10:47.441971+0000","last_clean":"2024-09-16T01:10:47.441971+0000","last_became_active":"2024-09-16T01:08:28.731581+0000","last_became_peered":"2024-09-16T01:08:28.731581+0000","last_unstale":"2024-09-16T01:10:47.441971+0000","last_undegraded":"2024-09-16T01:10:47.441971+0000","last_fullsized":"2024-09-16T01:10:47.441971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:18.690993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:51.277292+0000","last_change":"2024-09-16T01:08:26.713168+0000","last_active":"2024-09-16T01:10:51.277292+0000","last_peered":"2024-09-16T01:10:51.277292+0000","last_clean":"2024-09-16T01:10:51.277292+0000","last_became_active":"2024-09-16T01:08:26.712739+0000","last_became_peered":"2024-09-16T01:08:26.712739+0000","last_unstale":"2024-09-16T01:10:51.277292+0000","last_undegraded":"2024-09-16T01:10:51.277292+0000","last_fullsized":"2024-09-16T01:10:51.277292+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:21:33.114774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287358+0000","last_change":"2024-09-16T01:08:35.770828+0000","last_active":"2024-09-16T01:10:47.287358+0000","last_peered":"2024-09-16T01:10:47.287358+0000","last_clean":"2024-09-16T01:10:47.287358+0000","last_became_active":"2024-09-16T01:08:35.769401+0000","last_became_peered":"2024-09-16T01:08:35.769401+0000","last_unstale":"2024-09-16T01:10:47.287358+0000","last_undegraded":"2024-09-16T01:10:47.287358+0000","last_fullsized":"2024-09-16T01:10:47.287358+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:25.782440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442204+0000","last_change":"2024-09-16T01:08:28.726672+0000","last_active":"2024-09-16T01:10:47.442204+0000","last_peered":"2024-09-16T01:10:47.442204+0000","last_clean":"2024-09-16T01:10:47.442204+0000","last_became_active":"2024-09-16T01:08:28.726302+0000","last_became_peered":"2024-09-16T01:08:28.726302+0000","last_unstale":"2024-09-16T01:10:47.442204+0000","last_undegraded":"2024-09-16T01:10:47.442204+0000","last_fullsized":"2024-09-16T01:10:47.442204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:33:51.803894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268920+0000","last_change":"2024-09-16T01:08:26.711601+0000","last_active":"2024-09-16T01:10:47.268920+0000","last_peered":"2024-09-16T01:10:47.268920+0000","last_clean":"2024-09-16T01:10:47.268920+0000","last_became_active":"2024-09-16T01:08:26.711243+0000","last_became_peered":"2024-09-16T01:08:26.711243+0000","last_unstale":"2024-09-16T01:10:47.268920+0000","last_undegraded":"2024-09-16T01:10:47.268920+0000","last_fullsized":"2024-09-16T01:10:47.268920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:01.717290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525644+0000","last_change":"2024-09-16T01:08:35.773749+0000","last_active":"2024-09-16T01:10:47.525644+0000","last_peered":"2024-09-16T01:10:47.525644+0000","last_clean":"2024-09-16T01:10:47.525644+0000","last_became_active":"2024-09-16T01:08:35.773287+0000","last_became_peered":"2024-09-16T01:08:35.773287+0000","last_unstale":"2024-09-16T01:10:47.525644+0000","last_undegraded":"2024-09-16T01:10:47.525644+0000","last_fullsized":"2024-09-16T01:10:47.525644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:42.165800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271305+0000","last_change":"2024-09-16T01:08:28.719429+0000","last_active":"2024-09-16T01:10:47.271305+0000","last_peered":"2024-09-16T01:10:47.271305+0000","last_clean":"2024-09-16T01:10:47.271305+0000","last_became_active":"2024-09-16T01:08:28.719072+0000","last_became_peered":"2024-09-16T01:08:28.719072+0000","last_unstale":"2024-09-16T01:10:47.271305+0000","last_undegraded":"2024-09-16T01:10:47.271305+0000","last_fullsized":"2024-09-16T01:10:47.271305+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:53.169427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525622+0000","last_change":"2024-09-16T01:10:40.105562+0000","last_active":"2024-09-16T01:10:47.525622+0000","last_peered":"2024-09-16T01:10:47.525622+0000","last_clean":"2024-09-16T01:10:47.525622+0000","last_became_active":"2024-09-16T01:10:40.097568+0000","last_became_peered":"2024-09-16T01:10:40.097568+0000","last_unstale":"2024-09-16T01:10:47.525622+0000","last_undegraded":"2024-09-16T01:10:47.525622+0000","last_fullsized":"2024-09-16T01:10:47.525622+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:18.523898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268993+0000","last_change":"2024-09-16T01:08:35.775371+0000","last_active":"2024-09-16T01:10:47.268993+0000","last_peered":"2024-09-16T01:10:47.268993+0000","last_clean":"2024-09-16T01:10:47.268993+0000","last_became_active":"2024-09-16T01:08:35.774792+0000","last_became_peered":"2024-09-16T01:08:35.774792+0000","last_unstale":"2024-09-16T01:10:47.268993+0000","last_undegraded":"2024-09-16T01:10:47.268993+0000","last_fullsized":"2024-09-16T01:10:47.268993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:04.784295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289947+0000","last_change":"2024-09-16T01:08:28.721139+0000","last_active":"2024-09-16T01:10:47.289947+0000","last_peered":"2024-09-16T01:10:47.289947+0000","last_clean":"2024-09-16T01:10:47.289947+0000","last_became_active":"2024-09-16T01:08:28.720798+0000","last_became_peered":"2024-09-16T01:08:28.720798+0000","last_unstale":"2024-09-16T01:10:47.289947+0000","last_undegraded":"2024-09-16T01:10:47.289947+0000","last_fullsized":"2024-09-16T01:10:47.289947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:10.084416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":307,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:52.275701+0000","last_change":"2024-09-16T01:08:26.710274+0000","last_active":"2024-09-16T01:10:52.275701+0000","last_peered":"2024-09-16T01:10:52.275701+0000","last_clean":"2024-09-16T01:10:52.275701+0000","last_became_active":"2024-09-16T01:08:26.709726+0000","last_became_peered":"2024-09-16T01:08:26.709726+0000","last_unstale":"2024-09-16T01:10:52.275701+0000","last_undegraded":"2024-09-16T01:10:52.275701+0000","last_fullsized":"2024-09-16T01:10:52.275701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:50:05.004120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71705,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":226,"num_read_kb":181,"num_write":41,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286615+0000","last_change":"2024-09-16T01:09:39.121127+0000","last_active":"2024-09-16T01:10:47.286615+0000","last_peered":"2024-09-16T01:10:47.286615+0000","last_clean":"2024-09-16T01:10:47.286615+0000","last_became_active":"2024-09-16T01:09:39.120736+0000","last_became_peered":"2024-09-16T01:09:39.120736+0000","last_unstale":"2024-09-16T01:10:47.286615+0000","last_undegraded":"2024-09-16T01:10:47.286615+0000","last_fullsized":"2024-09-16T01:10:47.286615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:28.281410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438571+0000","last_change":"2024-09-16T01:09:39.168566+0000","last_active":"2024-09-16T01:10:47.438571+0000","last_peered":"2024-09-16T01:10:47.438571+0000","last_clean":"2024-09-16T01:10:47.438571+0000","last_became_active":"2024-09-16T01:09:39.168127+0000","last_became_peered":"2024-09-16T01:09:39.168127+0000","last_unstale":"2024-09-16T01:10:47.438571+0000","last_undegraded":"2024-09-16T01:10:47.438571+0000","last_fullsized":"2024-09-16T01:10:47.438571+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:52.054331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269209+0000","last_change":"2024-09-16T01:08:26.955804+0000","last_active":"2024-09-16T01:10:47.269209+0000","last_peered":"2024-09-16T01:10:47.269209+0000","last_clean":"2024-09-16T01:10:47.269209+0000","last_became_active":"2024-09-16T01:08:26.955445+0000","last_became_peered":"2024-09-16T01:08:26.955445+0000","last_unstale":"2024-09-16T01:10:47.269209+0000","last_undegraded":"2024-09-16T01:10:47.269209+0000","last_fullsized":"2024-09-16T01:10:47.269209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:55:54.226783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198541+0000","last_change":"2024-09-16T01:09:39.121335+0000","last_active":"2024-09-16T01:10:47.198541+0000","last_peered":"2024-09-16T01:10:47.198541+0000","last_clean":"2024-09-16T01:10:47.198541+0000","last_became_active":"2024-09-16T01:09:39.120738+0000","last_became_peered":"2024-09-16T01:09:39.120738+0000","last_unstale":"2024-09-16T01:10:47.198541+0000","last_undegraded":"2024-09-16T01:10:47.198541+0000","last_fullsized":"2024-09-16T01:10:47.198541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:09.085721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526309+0000","last_change":"2024-09-16T01:08:28.724768+0000","last_active":"2024-09-16T01:10:47.526309+0000","last_peered":"2024-09-16T01:10:47.526309+0000","last_clean":"2024-09-16T01:10:47.526309+0000","last_became_active":"2024-09-16T01:08:28.724421+0000","last_became_peered":"2024-09-16T01:08:28.724421+0000","last_unstale":"2024-09-16T01:10:47.526309+0000","last_undegraded":"2024-09-16T01:10:47.526309+0000","last_fullsized":"2024-09-16T01:10:47.526309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:01:12.399217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442273+0000","last_change":"2024-09-16T01:08:26.714398+0000","last_active":"2024-09-16T01:10:47.442273+0000","last_peered":"2024-09-16T01:10:47.442273+0000","last_clean":"2024-09-16T01:10:47.442273+0000","last_became_active":"2024-09-16T01:08:26.714027+0000","last_became_peered":"2024-09-16T01:08:26.714027+0000","last_unstale":"2024-09-16T01:10:47.442273+0000","last_undegraded":"2024-09-16T01:10:47.442273+0000","last_fullsized":"2024-09-16T01:10:47.442273+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:47.501203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286303+0000","last_change":"2024-09-16T01:08:35.768386+0000","last_active":"2024-09-16T01:10:47.286303+0000","last_peered":"2024-09-16T01:10:47.286303+0000","last_clean":"2024-09-16T01:10:47.286303+0000","last_became_active":"2024-09-16T01:08:35.767989+0000","last_became_peered":"2024-09-16T01:08:35.767989+0000","last_unstale":"2024-09-16T01:10:47.286303+0000","last_undegraded":"2024-09-16T01:10:47.286303+0000","last_fullsized":"2024-09-16T01:10:47.286303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:06.376767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438818+0000","last_change":"2024-09-16T01:08:28.717701+0000","last_active":"2024-09-16T01:10:47.438818+0000","last_peered":"2024-09-16T01:10:47.438818+0000","last_clean":"2024-09-16T01:10:47.438818+0000","last_became_active":"2024-09-16T01:08:28.717092+0000","last_became_peered":"2024-09-16T01:08:28.717092+0000","last_unstale":"2024-09-16T01:10:47.438818+0000","last_undegraded":"2024-09-16T01:10:47.438818+0000","last_fullsized":"2024-09-16T01:10:47.438818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:23.256803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"94'4","reported_seq":152,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:52.275173+0000","last_change":"2024-09-16T01:08:26.710144+0000","last_active":"2024-09-16T01:10:52.275173+0000","last_peered":"2024-09-16T01:10:52.275173+0000","last_clean":"2024-09-16T01:10:52.275173+0000","last_became_active":"2024-09-16T01:08:26.709773+0000","last_became_peered":"2024-09-16T01:08:26.709773+0000","last_unstale":"2024-09-16T01:10:52.275173+0000","last_undegraded":"2024-09-16T01:10:52.275173+0000","last_fullsized":"2024-09-16T01:10:52.275173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197960+0000","last_change":"2024-09-16T01:08:35.772740+0000","last_active":"2024-09-16T01:10:47.197960+0000","last_peered":"2024-09-16T01:10:47.197960+0000","last_clean":"2024-09-16T01:10:47.197960+0000","last_became_active":"2024-09-16T01:08:35.772344+0000","last_became_peered":"2024-09-16T01:08:35.772344+0000","last_unstale":"2024-09-16T01:10:47.197960+0000","last_undegraded":"2024-09-16T01:10:47.197960+0000","last_fullsized":"2024-09-16T01:10:47.197960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:11.556046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526102+0000","last_change":"2024-09-16T01:08:35.774831+0000","last_active":"2024-09-16T01:10:47.526102+0000","last_peered":"2024-09-16T01:10:47.526102+0000","last_clean":"2024-09-16T01:10:47.526102+0000","last_became_active":"2024-09-16T01:08:35.773881+0000","last_became_peered":"2024-09-16T01:08:35.773881+0000","last_unstale":"2024-09-16T01:10:47.526102+0000","last_undegraded":"2024-09-16T01:10:47.526102+0000","last_fullsized":"2024-09-16T01:10:47.526102+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:44.363650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.286613+0000","last_change":"2024-09-16T01:08:28.716025+0000","last_active":"2024-09-16T01:10:47.286613+0000","last_peered":"2024-09-16T01:10:47.286613+0000","last_clean":"2024-09-16T01:10:47.286613+0000","last_became_active":"2024-09-16T01:08:28.715712+0000","last_became_peered":"2024-09-16T01:08:28.715712+0000","last_unstale":"2024-09-16T01:10:47.286613+0000","last_undegraded":"2024-09-16T01:10:47.286613+0000","last_fullsized":"2024-09-16T01:10:47.286613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:27.910411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269391+0000","last_change":"2024-09-16T01:08:26.712822+0000","last_active":"2024-09-16T01:10:47.269391+0000","last_peered":"2024-09-16T01:10:47.269391+0000","last_clean":"2024-09-16T01:10:47.269391+0000","last_became_active":"2024-09-16T01:08:26.712280+0000","last_became_peered":"2024-09-16T01:08:26.712280+0000","last_unstale":"2024-09-16T01:10:47.269391+0000","last_undegraded":"2024-09-16T01:10:47.269391+0000","last_fullsized":"2024-09-16T01:10:47.269391+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:23.978492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525752+0000","last_change":"2024-09-16T01:08:35.772160+0000","last_active":"2024-09-16T01:10:47.525752+0000","last_peered":"2024-09-16T01:10:47.525752+0000","last_clean":"2024-09-16T01:10:47.525752+0000","last_became_active":"2024-09-16T01:08:35.771775+0000","last_became_peered":"2024-09-16T01:08:35.771775+0000","last_unstale":"2024-09-16T01:10:47.525752+0000","last_undegraded":"2024-09-16T01:10:47.525752+0000","last_fullsized":"2024-09-16T01:10:47.525752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:53.110851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197897+0000","last_change":"2024-09-16T01:08:28.717645+0000","last_active":"2024-09-16T01:10:47.197897+0000","last_peered":"2024-09-16T01:10:47.197897+0000","last_clean":"2024-09-16T01:10:47.197897+0000","last_became_active":"2024-09-16T01:08:28.717311+0000","last_became_peered":"2024-09-16T01:08:28.717311+0000","last_unstale":"2024-09-16T01:10:47.197897+0000","last_undegraded":"2024-09-16T01:10:47.197897+0000","last_fullsized":"2024-09-16T01:10:47.197897+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:55.579940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271582+0000","last_change":"2024-09-16T01:08:26.707568+0000","last_active":"2024-09-16T01:10:47.271582+0000","last_peered":"2024-09-16T01:10:47.271582+0000","last_clean":"2024-09-16T01:10:47.271582+0000","last_became_active":"2024-09-16T01:08:26.707274+0000","last_became_peered":"2024-09-16T01:08:26.707274+0000","last_unstale":"2024-09-16T01:10:47.271582+0000","last_undegraded":"2024-09-16T01:10:47.271582+0000","last_fullsized":"2024-09-16T01:10:47.271582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:29.914353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266595+0000","last_change":"2024-09-16T01:08:26.708632+0000","last_active":"2024-09-16T01:10:47.266595+0000","last_peered":"2024-09-16T01:10:47.266595+0000","last_clean":"2024-09-16T01:10:47.266595+0000","last_became_active":"2024-09-16T01:08:26.708174+0000","last_became_peered":"2024-09-16T01:08:26.708174+0000","last_unstale":"2024-09-16T01:10:47.266595+0000","last_undegraded":"2024-09-16T01:10:47.266595+0000","last_fullsized":"2024-09-16T01:10:47.266595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:43.589721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.269140+0000","last_change":"2024-09-16T01:08:28.727858+0000","last_active":"2024-09-16T01:10:47.269140+0000","last_peered":"2024-09-16T01:10:47.269140+0000","last_clean":"2024-09-16T01:10:47.269140+0000","last_became_active":"2024-09-16T01:08:28.727189+0000","last_became_peered":"2024-09-16T01:08:28.727189+0000","last_unstale":"2024-09-16T01:10:47.269140+0000","last_undegraded":"2024-09-16T01:10:47.269140+0000","last_fullsized":"2024-09-16T01:10:47.269140+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:01.156450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201517+0000","last_change":"2024-09-16T01:08:35.775024+0000","last_active":"2024-09-16T01:10:47.201517+0000","last_peered":"2024-09-16T01:10:47.201517+0000","last_clean":"2024-09-16T01:10:47.201517+0000","last_became_active":"2024-09-16T01:08:35.774583+0000","last_became_peered":"2024-09-16T01:08:35.774583+0000","last_unstale":"2024-09-16T01:10:47.201517+0000","last_undegraded":"2024-09-16T01:10:47.201517+0000","last_fullsized":"2024-09-16T01:10:47.201517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:22.944973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:51.274451+0000","last_change":"2024-09-16T01:08:26.712853+0000","last_active":"2024-09-16T01:10:51.274451+0000","last_peered":"2024-09-16T01:10:51.274451+0000","last_clean":"2024-09-16T01:10:51.274451+0000","last_became_active":"2024-09-16T01:08:26.712305+0000","last_became_peered":"2024-09-16T01:08:26.712305+0000","last_unstale":"2024-09-16T01:10:51.274451+0000","last_undegraded":"2024-09-16T01:10:51.274451+0000","last_fullsized":"2024-09-16T01:10:51.274451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:25.685034+0000","last_clean_scrub_stamp":"2024-09-16T01:08:25.685034+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:16.501448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289678+0000","last_change":"2024-09-16T01:08:28.713613+0000","last_active":"2024-09-16T01:10:47.289678+0000","last_peered":"2024-09-16T01:10:47.289678+0000","last_clean":"2024-09-16T01:10:47.289678+0000","last_became_active":"2024-09-16T01:08:28.712257+0000","last_became_peered":"2024-09-16T01:08:28.712257+0000","last_unstale":"2024-09-16T01:10:47.289678+0000","last_undegraded":"2024-09-16T01:10:47.289678+0000","last_fullsized":"2024-09-16T01:10:47.289678+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:40.970640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201763+0000","last_change":"2024-09-16T01:08:35.770829+0000","last_active":"2024-09-16T01:10:47.201763+0000","last_peered":"2024-09-16T01:10:47.201763+0000","last_clean":"2024-09-16T01:10:47.201763+0000","last_became_active":"2024-09-16T01:08:35.770430+0000","last_became_peered":"2024-09-16T01:08:35.770430+0000","last_unstale":"2024-09-16T01:10:47.201763+0000","last_undegraded":"2024-09-16T01:10:47.201763+0000","last_fullsized":"2024-09-16T01:10:47.201763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:34.745764+0000","last_clean_scrub_stamp":"2024-09-16T01:08:34.745764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:54.440495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525500+0000","last_change":"2024-09-16T01:09:39.119041+0000","last_active":"2024-09-16T01:10:47.525500+0000","last_peered":"2024-09-16T01:10:47.525500+0000","last_clean":"2024-09-16T01:10:47.525500+0000","last_became_active":"2024-09-16T01:09:39.118531+0000","last_became_peered":"2024-09-16T01:09:39.118531+0000","last_unstale":"2024-09-16T01:10:47.525500+0000","last_undegraded":"2024-09-16T01:10:47.525500+0000","last_fullsized":"2024-09-16T01:10:47.525500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:32:18.412355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201485+0000","last_change":"2024-09-16T01:08:28.730080+0000","last_active":"2024-09-16T01:10:47.201485+0000","last_peered":"2024-09-16T01:10:47.201485+0000","last_clean":"2024-09-16T01:10:47.201485+0000","last_became_active":"2024-09-16T01:08:28.729706+0000","last_became_peered":"2024-09-16T01:08:28.729706+0000","last_unstale":"2024-09-16T01:10:47.201485+0000","last_undegraded":"2024-09-16T01:10:47.201485+0000","last_fullsized":"2024-09-16T01:10:47.201485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:29.864871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201337+0000","last_change":"2024-09-16T01:08:28.729329+0000","last_active":"2024-09-16T01:10:47.201337+0000","last_peered":"2024-09-16T01:10:47.201337+0000","last_clean":"2024-09-16T01:10:47.201337+0000","last_became_active":"2024-09-16T01:08:28.728565+0000","last_became_peered":"2024-09-16T01:08:28.728565+0000","last_unstale":"2024-09-16T01:10:47.201337+0000","last_undegraded":"2024-09-16T01:10:47.201337+0000","last_fullsized":"2024-09-16T01:10:47.201337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:30.473925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525902+0000","last_change":"2024-09-16T01:08:28.723607+0000","last_active":"2024-09-16T01:10:47.525902+0000","last_peered":"2024-09-16T01:10:47.525902+0000","last_clean":"2024-09-16T01:10:47.525902+0000","last_became_active":"2024-09-16T01:08:28.723318+0000","last_became_peered":"2024-09-16T01:08:28.723318+0000","last_unstale":"2024-09-16T01:10:47.525902+0000","last_undegraded":"2024-09-16T01:10:47.525902+0000","last_fullsized":"2024-09-16T01:10:47.525902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:54:54.436002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271674+0000","last_change":"2024-09-16T01:08:28.725109+0000","last_active":"2024-09-16T01:10:47.271674+0000","last_peered":"2024-09-16T01:10:47.271674+0000","last_clean":"2024-09-16T01:10:47.271674+0000","last_became_active":"2024-09-16T01:08:28.724839+0000","last_became_peered":"2024-09-16T01:08:28.724839+0000","last_unstale":"2024-09-16T01:10:47.271674+0000","last_undegraded":"2024-09-16T01:10:47.271674+0000","last_fullsized":"2024-09-16T01:10:47.271674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:05.784546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525941+0000","last_change":"2024-09-16T01:08:28.710106+0000","last_active":"2024-09-16T01:10:47.525941+0000","last_peered":"2024-09-16T01:10:47.525941+0000","last_clean":"2024-09-16T01:10:47.525941+0000","last_became_active":"2024-09-16T01:08:28.709719+0000","last_became_peered":"2024-09-16T01:08:28.709719+0000","last_unstale":"2024-09-16T01:10:47.525941+0000","last_undegraded":"2024-09-16T01:10:47.525941+0000","last_fullsized":"2024-09-16T01:10:47.525941+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:35.737870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201759+0000","last_change":"2024-09-16T01:08:28.723020+0000","last_active":"2024-09-16T01:10:47.201759+0000","last_peered":"2024-09-16T01:10:47.201759+0000","last_clean":"2024-09-16T01:10:47.201759+0000","last_became_active":"2024-09-16T01:08:28.722611+0000","last_became_peered":"2024-09-16T01:08:28.722611+0000","last_unstale":"2024-09-16T01:10:47.201759+0000","last_undegraded":"2024-09-16T01:10:47.201759+0000","last_fullsized":"2024-09-16T01:10:47.201759+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:55:58.120041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198856+0000","last_change":"2024-09-16T01:08:28.724351+0000","last_active":"2024-09-16T01:10:47.198856+0000","last_peered":"2024-09-16T01:10:47.198856+0000","last_clean":"2024-09-16T01:10:47.198856+0000","last_became_active":"2024-09-16T01:08:28.724042+0000","last_became_peered":"2024-09-16T01:08:28.724042+0000","last_unstale":"2024-09-16T01:10:47.198856+0000","last_undegraded":"2024-09-16T01:10:47.198856+0000","last_fullsized":"2024-09-16T01:10:47.198856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:06.651725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.268742+0000","last_change":"2024-09-16T01:09:39.121080+0000","last_active":"2024-09-16T01:10:47.268742+0000","last_peered":"2024-09-16T01:10:47.268742+0000","last_clean":"2024-09-16T01:10:47.268742+0000","last_became_active":"2024-09-16T01:09:39.120707+0000","last_became_peered":"2024-09-16T01:09:39.120707+0000","last_unstale":"2024-09-16T01:10:47.268742+0000","last_undegraded":"2024-09-16T01:10:47.268742+0000","last_fullsized":"2024-09-16T01:10:47.268742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:24:08.521558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198060+0000","last_change":"2024-09-16T01:08:28.726328+0000","last_active":"2024-09-16T01:10:47.198060+0000","last_peered":"2024-09-16T01:10:47.198060+0000","last_clean":"2024-09-16T01:10:47.198060+0000","last_became_active":"2024-09-16T01:08:28.726013+0000","last_became_peered":"2024-09-16T01:08:28.726013+0000","last_unstale":"2024-09-16T01:10:47.198060+0000","last_undegraded":"2024-09-16T01:10:47.198060+0000","last_fullsized":"2024-09-16T01:10:47.198060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:15.208120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438556+0000","last_change":"2024-09-16T01:08:28.719731+0000","last_active":"2024-09-16T01:10:47.438556+0000","last_peered":"2024-09-16T01:10:47.438556+0000","last_clean":"2024-09-16T01:10:47.438556+0000","last_became_active":"2024-09-16T01:08:28.719368+0000","last_became_peered":"2024-09-16T01:08:28.719368+0000","last_unstale":"2024-09-16T01:10:47.438556+0000","last_undegraded":"2024-09-16T01:10:47.438556+0000","last_fullsized":"2024-09-16T01:10:47.438556+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:27:49.096832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271474+0000","last_change":"2024-09-16T01:08:28.719407+0000","last_active":"2024-09-16T01:10:47.271474+0000","last_peered":"2024-09-16T01:10:47.271474+0000","last_clean":"2024-09-16T01:10:47.271474+0000","last_became_active":"2024-09-16T01:08:28.719131+0000","last_became_peered":"2024-09-16T01:08:28.719131+0000","last_unstale":"2024-09-16T01:10:47.271474+0000","last_undegraded":"2024-09-16T01:10:47.271474+0000","last_fullsized":"2024-09-16T01:10:47.271474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:19:16.364891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525822+0000","last_change":"2024-09-16T01:08:28.717282+0000","last_active":"2024-09-16T01:10:47.525822+0000","last_peered":"2024-09-16T01:10:47.525822+0000","last_clean":"2024-09-16T01:10:47.525822+0000","last_became_active":"2024-09-16T01:08:28.716971+0000","last_became_peered":"2024-09-16T01:08:28.716971+0000","last_unstale":"2024-09-16T01:10:47.525822+0000","last_undegraded":"2024-09-16T01:10:47.525822+0000","last_fullsized":"2024-09-16T01:10:47.525822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:30.485784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.201931+0000","last_change":"2024-09-16T01:08:28.722870+0000","last_active":"2024-09-16T01:10:47.201931+0000","last_peered":"2024-09-16T01:10:47.201931+0000","last_clean":"2024-09-16T01:10:47.201931+0000","last_became_active":"2024-09-16T01:08:28.720656+0000","last_became_peered":"2024-09-16T01:08:28.720656+0000","last_unstale":"2024-09-16T01:10:47.201931+0000","last_undegraded":"2024-09-16T01:10:47.201931+0000","last_fullsized":"2024-09-16T01:10:47.201931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:59:39.065802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.287352+0000","last_change":"2024-09-16T01:08:28.719771+0000","last_active":"2024-09-16T01:10:47.287352+0000","last_peered":"2024-09-16T01:10:47.287352+0000","last_clean":"2024-09-16T01:10:47.287352+0000","last_became_active":"2024-09-16T01:08:28.719480+0000","last_became_peered":"2024-09-16T01:08:28.719480+0000","last_unstale":"2024-09-16T01:10:47.287352+0000","last_undegraded":"2024-09-16T01:10:47.287352+0000","last_fullsized":"2024-09-16T01:10:47.287352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:04.999376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525997+0000","last_change":"2024-09-16T01:08:28.715525+0000","last_active":"2024-09-16T01:10:47.525997+0000","last_peered":"2024-09-16T01:10:47.525997+0000","last_clean":"2024-09-16T01:10:47.525997+0000","last_became_active":"2024-09-16T01:08:28.715193+0000","last_became_peered":"2024-09-16T01:08:28.715193+0000","last_unstale":"2024-09-16T01:10:47.525997+0000","last_undegraded":"2024-09-16T01:10:47.525997+0000","last_fullsized":"2024-09-16T01:10:47.525997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:31.087506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.438182+0000","last_change":"2024-09-16T01:08:28.720104+0000","last_active":"2024-09-16T01:10:47.438182+0000","last_peered":"2024-09-16T01:10:47.438182+0000","last_clean":"2024-09-16T01:10:47.438182+0000","last_became_active":"2024-09-16T01:08:28.719791+0000","last_became_peered":"2024-09-16T01:08:28.719791+0000","last_unstale":"2024-09-16T01:10:47.438182+0000","last_undegraded":"2024-09-16T01:10:47.438182+0000","last_fullsized":"2024-09-16T01:10:47.438182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:49:00.027819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525723+0000","last_change":"2024-09-16T01:08:28.726164+0000","last_active":"2024-09-16T01:10:47.525723+0000","last_peered":"2024-09-16T01:10:47.525723+0000","last_clean":"2024-09-16T01:10:47.525723+0000","last_became_active":"2024-09-16T01:08:28.725842+0000","last_became_peered":"2024-09-16T01:08:28.725842+0000","last_unstale":"2024-09-16T01:10:47.525723+0000","last_undegraded":"2024-09-16T01:10:47.525723+0000","last_fullsized":"2024-09-16T01:10:47.525723+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:25:09.011153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525751+0000","last_change":"2024-09-16T01:08:28.717768+0000","last_active":"2024-09-16T01:10:47.525751+0000","last_peered":"2024-09-16T01:10:47.525751+0000","last_clean":"2024-09-16T01:10:47.525751+0000","last_became_active":"2024-09-16T01:08:28.717457+0000","last_became_peered":"2024-09-16T01:08:28.717457+0000","last_unstale":"2024-09-16T01:10:47.525751+0000","last_undegraded":"2024-09-16T01:10:47.525751+0000","last_fullsized":"2024-09-16T01:10:47.525751+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:32.705314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.271427+0000","last_change":"2024-09-16T01:08:28.727468+0000","last_active":"2024-09-16T01:10:47.271427+0000","last_peered":"2024-09-16T01:10:47.271427+0000","last_clean":"2024-09-16T01:10:47.271427+0000","last_became_active":"2024-09-16T01:08:28.726863+0000","last_became_peered":"2024-09-16T01:08:28.726863+0000","last_unstale":"2024-09-16T01:10:47.271427+0000","last_undegraded":"2024-09-16T01:10:47.271427+0000","last_fullsized":"2024-09-16T01:10:47.271427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:44:09.691053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198700+0000","last_change":"2024-09-16T01:08:28.720748+0000","last_active":"2024-09-16T01:10:47.198700+0000","last_peered":"2024-09-16T01:10:47.198700+0000","last_clean":"2024-09-16T01:10:47.198700+0000","last_became_active":"2024-09-16T01:08:28.718203+0000","last_became_peered":"2024-09-16T01:08:28.718203+0000","last_unstale":"2024-09-16T01:10:47.198700+0000","last_undegraded":"2024-09-16T01:10:47.198700+0000","last_fullsized":"2024-09-16T01:10:47.198700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:20.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442364+0000","last_change":"2024-09-16T01:08:28.724001+0000","last_active":"2024-09-16T01:10:47.442364+0000","last_peered":"2024-09-16T01:10:47.442364+0000","last_clean":"2024-09-16T01:10:47.442364+0000","last_became_active":"2024-09-16T01:08:28.723673+0000","last_became_peered":"2024-09-16T01:08:28.723673+0000","last_unstale":"2024-09-16T01:10:47.442364+0000","last_undegraded":"2024-09-16T01:10:47.442364+0000","last_fullsized":"2024-09-16T01:10:47.442364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:36.044104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266799+0000","last_change":"2024-09-16T01:08:28.725006+0000","last_active":"2024-09-16T01:10:47.266799+0000","last_peered":"2024-09-16T01:10:47.266799+0000","last_clean":"2024-09-16T01:10:47.266799+0000","last_became_active":"2024-09-16T01:08:28.724712+0000","last_became_peered":"2024-09-16T01:08:28.724712+0000","last_unstale":"2024-09-16T01:10:47.266799+0000","last_undegraded":"2024-09-16T01:10:47.266799+0000","last_fullsized":"2024-09-16T01:10:47.266799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:38.620747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.439065+0000","last_change":"2024-09-16T01:08:28.727126+0000","last_active":"2024-09-16T01:10:47.439065+0000","last_peered":"2024-09-16T01:10:47.439065+0000","last_clean":"2024-09-16T01:10:47.439065+0000","last_became_active":"2024-09-16T01:08:28.726828+0000","last_became_peered":"2024-09-16T01:08:28.726828+0000","last_unstale":"2024-09-16T01:10:47.439065+0000","last_undegraded":"2024-09-16T01:10:47.439065+0000","last_fullsized":"2024-09-16T01:10:47.439065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:05.183921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.197771+0000","last_change":"2024-09-16T01:08:28.731542+0000","last_active":"2024-09-16T01:10:47.197771+0000","last_peered":"2024-09-16T01:10:47.197771+0000","last_clean":"2024-09-16T01:10:47.197771+0000","last_became_active":"2024-09-16T01:08:28.731209+0000","last_became_peered":"2024-09-16T01:08:28.731209+0000","last_unstale":"2024-09-16T01:10:47.197771+0000","last_undegraded":"2024-09-16T01:10:47.197771+0000","last_fullsized":"2024-09-16T01:10:47.197771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:09.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.442000+0000","last_change":"2024-09-16T01:08:28.731901+0000","last_active":"2024-09-16T01:10:47.442000+0000","last_peered":"2024-09-16T01:10:47.442000+0000","last_clean":"2024-09-16T01:10:47.442000+0000","last_became_active":"2024-09-16T01:08:28.731589+0000","last_became_peered":"2024-09-16T01:08:28.731589+0000","last_unstale":"2024-09-16T01:10:47.442000+0000","last_undegraded":"2024-09-16T01:10:47.442000+0000","last_fullsized":"2024-09-16T01:10:47.442000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:22.638704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.266403+0000","last_change":"2024-09-16T01:08:28.724359+0000","last_active":"2024-09-16T01:10:47.266403+0000","last_peered":"2024-09-16T01:10:47.266403+0000","last_clean":"2024-09-16T01:10:47.266403+0000","last_became_active":"2024-09-16T01:08:28.724016+0000","last_became_peered":"2024-09-16T01:08:28.724016+0000","last_unstale":"2024-09-16T01:10:47.266403+0000","last_undegraded":"2024-09-16T01:10:47.266403+0000","last_fullsized":"2024-09-16T01:10:47.266403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:23.937631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.289667+0000","last_change":"2024-09-16T01:08:28.719044+0000","last_active":"2024-09-16T01:10:47.289667+0000","last_peered":"2024-09-16T01:10:47.289667+0000","last_clean":"2024-09-16T01:10:47.289667+0000","last_became_active":"2024-09-16T01:08:28.718673+0000","last_became_peered":"2024-09-16T01:08:28.718673+0000","last_unstale":"2024-09-16T01:10:47.289667+0000","last_undegraded":"2024-09-16T01:10:47.289667+0000","last_fullsized":"2024-09-16T01:10:47.289667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:01.246019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.198621+0000","last_change":"2024-09-16T01:08:28.725204+0000","last_active":"2024-09-16T01:10:47.198621+0000","last_peered":"2024-09-16T01:10:47.198621+0000","last_clean":"2024-09-16T01:10:47.198621+0000","last_became_active":"2024-09-16T01:08:28.724365+0000","last_became_peered":"2024-09-16T01:08:28.724365+0000","last_unstale":"2024-09-16T01:10:47.198621+0000","last_undegraded":"2024-09-16T01:10:47.198621+0000","last_fullsized":"2024-09-16T01:10:47.198621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:15.276624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525772+0000","last_change":"2024-09-16T01:08:28.729351+0000","last_active":"2024-09-16T01:10:47.525772+0000","last_peered":"2024-09-16T01:10:47.525772+0000","last_clean":"2024-09-16T01:10:47.525772+0000","last_became_active":"2024-09-16T01:08:28.729043+0000","last_became_peered":"2024-09-16T01:08:28.729043+0000","last_unstale":"2024-09-16T01:10:47.525772+0000","last_undegraded":"2024-09-16T01:10:47.525772+0000","last_fullsized":"2024-09-16T01:10:47.525772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:21.078202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.526118+0000","last_change":"2024-09-16T01:08:28.725987+0000","last_active":"2024-09-16T01:10:47.526118+0000","last_peered":"2024-09-16T01:10:47.526118+0000","last_clean":"2024-09-16T01:10:47.526118+0000","last_became_active":"2024-09-16T01:08:28.725620+0000","last_became_peered":"2024-09-16T01:08:28.725620+0000","last_unstale":"2024-09-16T01:10:47.526118+0000","last_undegraded":"2024-09-16T01:10:47.526118+0000","last_fullsized":"2024-09-16T01:10:47.526118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:50.991242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"91'8","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T01:10:47.525943+0000","last_change":"2024-09-16T01:08:28.717740+0000","last_active":"2024-09-16T01:10:47.525943+0000","last_peered":"2024-09-16T01:10:47.525943+0000","last_clean":"2024-09-16T01:10:47.525943+0000","last_became_active":"2024-09-16T01:08:28.716702+0000","last_became_peered":"2024-09-16T01:08:28.716702+0000","last_unstale":"2024-09-16T01:10:47.525943+0000","last_undegraded":"2024-09-16T01:10:47.525943+0000","last_fullsized":"2024-09-16T01:10:47.525943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:08:27.695778+0000","last_clean_scrub_stamp":"2024-09-16T01:08:27.695778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:44.855686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799177,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1022,"num_read_kb":727,"num_write":432,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":2397531,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":317,"ondisk_log_size":317,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107904,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481835,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1040384,"data_stored":537345,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":70,"seq":300647710767,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712764,"statfs":{"total":95995035648,"available":95961870336,"internally_reserved":0,"allocated":1232896,"data_stored":1523624,"data_compressed":83561,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":65,"seq":279172874292,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":934859,"data_compressed":51680,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103225,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":930683,"data_compressed":51680,"data_compressed_allocated":69632,"data_compressed_original":483328,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":1135399,"data_compressed":68359,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1032192,"data_stored":532723,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1155072,"data_stored":1457591,"data_compressed":83561,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":539113,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1114112,"data_stored":1254365,"data_compressed":66882,"data_compressed_allocated":90112,"data_compressed_original":790528,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":603808,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1110016,"data_stored":604338,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1056768,"data_stored":534111,"data_compressed":32391,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35968,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":71705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":73695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330494,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":403143,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:10:56.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:56.835 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:57.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:57 smithi078 ceph-mon[30937]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-09-16T01:10:57.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:57 smithi078 ceph-mon[30937]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/3889418066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 msgr2=0x7f7ff010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/3889418066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff010e790 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8009900 tx=0x7f7fe802a410 comp rx=0 tx=0).stop 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/3889418066 shutdown_connections 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/3889418066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff010e790 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/3889418066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/3889418066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 0x7f7ff01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/3889418066 >> 172.21.15.78:0/3889418066 conn(0x7f7ff00fc7d0 msgr2=0x7f7ff00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/3889418066 shutdown_connections 2024-09-16T01:10:57.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.314+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/3889418066 wait complete. 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 Processor -- start 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 -- start start 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 0x7f7ff01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0112110 con 0x7f7ff010c2f0 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0111f90 con 0x7f7ff0102df0 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7ff4ef1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0112290 con 0x7f7ff01037a0 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47802/0 (socket says 172.21.15.78:47802) 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 -- 172.21.15.78:0/4007785237 learned_addr learned my addr 172.21.15.78:0/4007785237 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:57.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7fee59c640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 0x7f7ff01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7fedd9b640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 -- 172.21.15.78:0/4007785237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 msgr2=0x7f7ff0198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff0198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 -- 172.21.15.78:0/4007785237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 msgr2=0x7f7ff01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 0x7f7ff01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7feed9d640 1 -- 172.21.15.78:0/4007785237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff019d570 con 0x7f7ff010c2f0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.315+0000 7f7fedd9b640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7feed9d640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8005ab0 tx=0x7f7fe80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe803f030 con 0x7f7ff010c2f0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe8003ce0 con 0x7f7ff010c2f0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff019d800 con 0x7f7ff010c2f0 2024-09-16T01:10:57.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8035070 con 0x7f7ff010c2f0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.316+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff01a5140 con 0x7f7ff010c2f0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.317+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff01031d0 con 0x7f7ff010c2f0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.318+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 24) ==== 138582+0+0 (secure 0 0 0) 0x7f7fe8042030 con 0x7f7ff010c2f0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.319+0000 7f7fdf7fe640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 0x7f7fd80a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.319+0000 7f7fee59c640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 0x7f7fd80a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.319+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7fe80f0d10 con 0x7f7ff010c2f0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.319+0000 7f7fee59c640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 0x7f7fd80a5a10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0005fd0 tx=0x7f7fe0009680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:57.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.322+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7fe80bd180 con 0x7f7ff010c2f0 2024-09-16T01:10:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:57 smithi148 ceph-mon[35030]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-09-16T01:10:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:57 smithi148 ceph-mon[35030]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:57.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.524+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ff0199920 con 0x7f7ff010c2f0 2024-09-16T01:10:57.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.525+0000 7f7fdf7fe640 1 -- 172.21.15.78:0/4007785237 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11918 (secure 0 0 0) 0x7f7fe80bca80 con 0x7f7ff010c2f0 2024-09-16T01:10:57.528 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:57.529 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"btime":"2024-09-16T01:10:45:984642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:10:45.934449+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:10:57.530 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-09-16T01:10:57.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 msgr2=0x7f7fd80a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:57.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 0x7f7fd80a5a10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0005fd0 tx=0x7f7fe0009680 comp rx=0 tx=0).stop 2024-09-16T01:10:57.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 msgr2=0x7f7ff019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8005ab0 tx=0x7f7fe80042f0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 shutdown_connections 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd80a3550 0x7f7fd80a5a10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff010c2f0 0x7f7ff019ce90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ff01037a0 0x7f7ff0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 --2- 172.21.15.78:0/4007785237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff0102df0 0x7f7ff01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:57.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.527+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 >> 172.21.15.78:0/4007785237 conn(0x7f7ff00fc7d0 msgr2=0x7f7ff00fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:57.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.528+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 shutdown_connections 2024-09-16T01:10:57.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:57.528+0000 7f7ff4ef1640 1 -- 172.21.15.78:0/4007785237 wait complete. 2024-09-16T01:10:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:57 smithi193 ceph-mon[35701]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-09-16T01:10:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:57 smithi193 ceph-mon[35701]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-16T01:10:58.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:58 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4007785237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:58.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:10:58.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:58 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4007785237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:10:58.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:58 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4007785237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:10:59.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 -- 172.21.15.78:0/2926500088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470100a80 msgr2=0x7f3470100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:59.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 --2- 172.21.15.78:0/2926500088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470100a80 0x7f3470100ee0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f345c009930 tx=0x7f345c02a420 comp rx=0 tx=0).stop 2024-09-16T01:10:59.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 -- 172.21.15.78:0/2926500088 shutdown_connections 2024-09-16T01:10:59.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 --2- 172.21.15.78:0/2926500088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 0x7f347010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 --2- 172.21.15.78:0/2926500088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470100a80 0x7f3470100ee0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 --2- 172.21.15.78:0/2926500088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470106a80 0x7f3470106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 -- 172.21.15.78:0/2926500088 >> 172.21.15.78:0/2926500088 conn(0x7f34700fc7a0 msgr2=0x7f34700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 -- 172.21.15.78:0/2926500088 shutdown_connections 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.991+0000 7f3476cfe640 1 -- 172.21.15.78:0/2926500088 wait complete. 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 Processor -- start 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 -- start start 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470100a80 0x7f34701970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 0x7f34701975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3470110e80 con 0x7f3470106a80 2024-09-16T01:10:59.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3470110d00 con 0x7f3470100a80 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3476cfe640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3470111000 con 0x7f3470101420 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f347529b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f347529b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47820/0 (socket says 172.21.15.78:47820) 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f347529b640 1 -- 172.21.15.78:0/1257082279 learned_addr learned my addr 172.21.15.78:0/1257082279 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f346ffff640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 0x7f34701975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.992+0000 7f3474a9a640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470100a80 0x7f34701970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 -- 172.21.15.78:0/1257082279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 msgr2=0x7f34701975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 0x7f34701975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 -- 172.21.15.78:0/1257082279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470100a80 msgr2=0x7f34701970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470100a80 0x7f34701970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 -- 172.21.15.78:0/1257082279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34701a6910 con 0x7f3470106a80 2024-09-16T01:10:59.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f347529b640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f346800eaa0 tx=0x7f346800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346800cdb0 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34701a6c00 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3468005020 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680116b0 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.993+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34701a7110 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.994+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3470100ee0 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.995+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 24) ==== 138582+0+0 (secure 0 0 0) 0x7f3468005560 con 0x7f3470106a80 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.996+0000 7f346dffb640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 0x7f34600a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:10:59.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.996+0000 7f3474a9a640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 0x7f34600a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:10:59.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.996+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f34680d04b0 con 0x7f3470106a80 2024-09-16T01:10:59.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:58.996+0000 7f3474a9a640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 0x7f34600a5990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34640058c0 tx=0x7f346400a2d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:10:59.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.000+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3468016030 con 0x7f3470106a80 2024-09-16T01:10:59.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.202+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f347002d7c0 con 0x7f3470106a80 2024-09-16T01:10:59.204 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:10:59 smithi078 ceph-mon[30937]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-09-16T01:10:59.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.203+0000 7f346dffb640 1 -- 172.21.15.78:0/1257082279 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11918 (secure 0 0 0) 0x7f346809c260 con 0x7f3470106a80 2024-09-16T01:10:59.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:10:59.205 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"btime":"2024-09-16T01:10:45:984642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:10:45.934449+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:10:59.205 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-09-16T01:10:59.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 msgr2=0x7f34600a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:59.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 0x7f34600a5990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34640058c0 tx=0x7f346400a2d0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 msgr2=0x7f3470197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f346800eaa0 tx=0x7f346800ef60 comp rx=0 tx=0).stop 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 shutdown_connections 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34600a34d0 0x7f34600a5990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3470106a80 0x7f3470197b20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3470101420 0x7f34701975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 --2- 172.21.15.78:0/1257082279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3470100a80 0x7f34701970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 >> 172.21.15.78:0/1257082279 conn(0x7f34700fc7a0 msgr2=0x7f34700fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 shutdown_connections 2024-09-16T01:10:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:10:59.205+0000 7f3476cfe640 1 -- 172.21.15.78:0/1257082279 wait complete. 2024-09-16T01:10:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:10:59 smithi148 ceph-mon[35030]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-09-16T01:10:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:10:59 smithi193 ceph-mon[35701]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 60 op/s 2024-09-16T01:10:59.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:11:00.103 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1257082279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:00.121 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:00 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1257082279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:00.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:00 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1257082279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:00.394 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:11:00.101+0000 7fda4c889640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:11:00.394 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:11:00.101+0000 7fda4c889640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:11:00.394 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:11:00.101+0000 7fda4c889640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:11:00.394 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:11:00.101+0000 7fda4c889640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:11:00.394 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:11:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:11:00.101+0000 7fda4c889640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T01:11:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 -- 172.21.15.78:0/1716989729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100480 msgr2=0x7fe37c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 --2- 172.21.15.78:0/1716989729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100480 0x7fe37c100860 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe364009900 tx=0x7fe36402a410 comp rx=0 tx=0).stop 2024-09-16T01:11:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 -- 172.21.15.78:0/1716989729 shutdown_connections 2024-09-16T01:11:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 --2- 172.21.15.78:0/1716989729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c1055b0 0x7fe37c10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 --2- 172.21.15.78:0/1716989729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c100e30 0x7fe37c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 --2- 172.21.15.78:0/1716989729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100480 0x7fe37c100860 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 -- 172.21.15.78:0/1716989729 >> 172.21.15.78:0/1716989729 conn(0x7fe37c077210 msgr2=0x7fe37c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 -- 172.21.15.78:0/1716989729 shutdown_connections 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 -- 172.21.15.78:0/1716989729 wait complete. 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.610+0000 7fe381200640 1 Processor -- start 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 -- start start 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c100480 0x7fe37c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 0x7fe37c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c110e50 con 0x7fe37c100e30 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c110cd0 con 0x7fe37c1055b0 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe381200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c110fd0 con 0x7fe37c100480 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47840/0 (socket says 172.21.15.78:47840) 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 -- 172.21.15.78:0/3882936644 learned_addr learned my addr 172.21.15.78:0/3882936644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37ad9d640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c100480 0x7fe37c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 -- 172.21.15.78:0/3882936644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c100480 msgr2=0x7fe37c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c100480 0x7fe37c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37b59e640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 0x7fe37c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 -- 172.21.15.78:0/3882936644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 msgr2=0x7fe37c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 0x7fe37c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 -- 172.21.15.78:0/3882936644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c19d400 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37b59e640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 0x7fe37c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.611+0000 7fe37a59c640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00d950 tx=0x7fe36c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.612+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c017070 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.612+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c19d6f0 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.612+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe36c009c30 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.612+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c00f6b0 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.612+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c1a4fd0 con 0x7fe37c100e30 2024-09-16T01:11:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.613+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c100860 con 0x7fe37c100e30 2024-09-16T01:11:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.614+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 24) ==== 138582+0+0 (secure 0 0 0) 0x7fe36c0109d0 con 0x7fe37c100e30 2024-09-16T01:11:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.615+0000 7fe373fff640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 0x7fe3680a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.615+0000 7fe37ad9d640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 0x7fe3680a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.615+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe36c0d0080 con 0x7fe37c100e30 2024-09-16T01:11:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.616+0000 7fe37ad9d640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 0x7fe3680a5990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe364005ab0 tx=0x7fe364005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.620+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe36c009de0 con 0x7fe37c100e30 2024-09-16T01:11:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.793+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe37c109800 con 0x7fe37c100e30 2024-09-16T01:11:00.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.794+0000 7fe373fff640 1 -- 172.21.15.78:0/3882936644 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25221 (secure 0 0 0) 0x7fe36c09c480 con 0x7fe37c100e30 2024-09-16T01:11:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:00.796 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:10:47.191895+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1816200719":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:0/557237582":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1671709190":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3382939699":"2024-09-17T01:10:47.191866+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:11:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 msgr2=0x7fe3680a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 0x7fe3680a5990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe364005ab0 tx=0x7fe364005a20 comp rx=0 tx=0).stop 2024-09-16T01:11:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 msgr2=0x7fe37c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00d950 tx=0x7fe36c00de10 comp rx=0 tx=0).stop 2024-09-16T01:11:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 shutdown_connections 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3680a34d0 0x7fe3680a5990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe37c1055b0 0x7fe37c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c100e30 0x7fe37c198990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 --2- 172.21.15.78:0/3882936644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe37c100480 0x7fe37c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 >> 172.21.15.78:0/3882936644 conn(0x7fe37c077210 msgr2=0x7fe37c10dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 shutdown_connections 2024-09-16T01:11:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:00.796+0000 7fe381200640 1 -- 172.21.15.78:0/3882936644 wait complete. 2024-09-16T01:11:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:01 smithi148 ceph-mon[35030]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-09-16T01:11:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:01 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3882936644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:11:01.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:01 smithi078 ceph-mon[30937]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-09-16T01:11:01.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:01 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3882936644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:11:01.509 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-16T01:11:01.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:01 smithi193 ceph-mon[35701]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-09-16T01:11:01.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:01 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3882936644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:11:01.754 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:02.984 INFO:teuthology.orchestra.run.smithi078.stdout:{"run_dir":"/var/run/ceph"} 2024-09-16T01:11:02.985 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-16T01:11:02.985 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3393 2024-09-16T01:11:02.985 DEBUG:tasks.quiescer:Yielding 2024-09-16T01:11:02.985 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T01:11:02.996 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T01:11:02.996 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:11:02.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-09-16T01:11:02.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:11:03.033 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:03.045 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem Size Used Avail Use% Mounted on 2024-09-16T01:11:03.045 INFO:teuthology.orchestra.run.smithi078.stdout:0@fa4731de-73c6-11ef-bceb-c7b262605968.cephfs=/volumes/qa/sv_1/d9b8d722-5b7a-4d90-86f4-84adec585113 1.1T 376M 1.1T 1% /tmp/cephfs 2024-09-16T01:11:03.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-09-16T01:11:03.137 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-09-16T01:11:03.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-09-16T01:11:03.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:03 smithi078 ceph-mon[30937]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-09-16T01:11:03.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:03 smithi078 ceph-mon[30937]: mgrmap e25: x(active, since 14s), standbys: y, z 2024-09-16T01:11:03.255 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:03 smithi148 ceph-mon[35030]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-09-16T01:11:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:03 smithi148 ceph-mon[35030]: mgrmap e25: x(active, since 14s), standbys: y, z 2024-09-16T01:11:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:03 smithi193 ceph-mon[35701]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-09-16T01:11:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:03 smithi193 ceph-mon[35701]: mgrmap e25: x(active, since 14s), standbys: y, z 2024-09-16T01:11:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 -- 172.21.15.78:0/2521368028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc102ef0 msgr2=0x7f27fc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:03.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 --2- 172.21.15.78:0/2521368028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc102ef0 0x7f27fc10da40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009ca0 tx=0x7f27f402a950 comp rx=0 tx=0).stop 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 -- 172.21.15.78:0/2521368028 shutdown_connections 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 --2- 172.21.15.78:0/2521368028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc102ef0 0x7f27fc10da40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 --2- 172.21.15.78:0/2521368028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 0x7f27fc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 --2- 172.21.15.78:0/2521368028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc100e20 0x7f27fc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 -- 172.21.15.78:0/2521368028 >> 172.21.15.78:0/2521368028 conn(0x7f27fc0fc7d0 msgr2=0x7f27fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 -- 172.21.15.78:0/2521368028 shutdown_connections 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.794+0000 7f2803bed640 1 -- 172.21.15.78:0/2521368028 wait complete. 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.795+0000 7f2803bed640 1 Processor -- start 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.795+0000 7f2803bed640 1 -- start start 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.795+0000 7f2803bed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.795+0000 7f2803bed640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 0x7f27fc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:03.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 0x7f27fc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc110f80 con 0x7f27fc100e20 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc110e00 con 0x7f27fc102ef0 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc111100 con 0x7f27fc102570 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47850/0 (socket says 172.21.15.78:47850) 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 -- 172.21.15.78:0/267046000 learned_addr learned my addr 172.21.15.78:0/267046000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f280218a640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 0x7f27fc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801188640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 0x7f27fc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 -- 172.21.15.78:0/267046000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 msgr2=0x7f27fc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 0x7f27fc0fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 -- 172.21.15.78:0/267046000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 msgr2=0x7f27fc0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:03.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 0x7f27fc0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 -- 172.21.15.78:0/267046000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc1a6880 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f280218a640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 0x7f27fc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2801989640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f27f800c920 tx=0x7f27f800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8007c70 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27fc1a6b70 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f8007e10 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8018890 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.796+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27fc1a7100 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.797+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27fc101200 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.798+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f27f8018a70 con 0x7f27fc100e20 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.799+0000 7f27f2ffd640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 0x7f27e80a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:03.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.799+0000 7f2801188640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 0x7f27e80a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:03.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.799+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f27f80d0620 con 0x7f27fc100e20 2024-09-16T01:11:03.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.800+0000 7f2801188640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 0x7f27e80a5a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0fe710 tx=0x7f27ec005c50 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:03.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:03.803+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27f8014030 con 0x7f27fc100e20 2024-09-16T01:11:04.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.014+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27fc10a0a0 con 0x7f27fc100e20 2024-09-16T01:11:04.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.015+0000 7f27f2ffd640 1 -- 172.21.15.78:0/267046000 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11918 (secure 0 0 0) 0x7f27f800de20 con 0x7f27fc100e20 2024-09-16T01:11:04.016 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:04.016 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"btime":"2024-09-16T01:10:45:984642+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:10:45.934449+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:11:04.016 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 msgr2=0x7f27e80a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 0x7f27e80a5a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0fe710 tx=0x7f27ec005c50 comp rx=0 tx=0).stop 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 msgr2=0x7f27fc0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f27f800c920 tx=0x7f27f800cde0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 shutdown_connections 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f27e80a3560 0x7f27e80a5a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27fc102ef0 0x7f27fc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27fc102570 0x7f27fc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 --2- 172.21.15.78:0/267046000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc100e20 0x7f27fc0fcc80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 >> 172.21.15.78:0/267046000 conn(0x7f27fc0fc7d0 msgr2=0x7f27fc06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 shutdown_connections 2024-09-16T01:11:04.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:04.018+0000 7f2803bed640 1 -- 172.21.15.78:0/267046000 wait complete. 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- 172.21.15.78:0/2379821493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38841016d0 msgr2=0x7f3884101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- 172.21.15.78:0/2379821493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38841016d0 0x7f3884101b10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3870009900 tx=0x7f387002a480 comp rx=0 tx=0).stop 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- 172.21.15.78:0/2379821493 shutdown_connections 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- 172.21.15.78:0/2379821493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884102050 0x7f388410d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- 172.21.15.78:0/2379821493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38841016d0 0x7f3884101b10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- 172.21.15.78:0/2379821493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3884105af0 0x7f3884105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- 172.21.15.78:0/2379821493 >> 172.21.15.78:0/2379821493 conn(0x7f3884075ec0 msgr2=0x7f38840762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- 172.21.15.78:0/2379821493 shutdown_connections 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- 172.21.15.78:0/2379821493 wait complete. 2024-09-16T01:11:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 Processor -- start 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 -- start start 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 0x7f3884198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.027+0000 7f388acd0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 0x7f388419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f388acd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110de0 con 0x7f3884102050 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f388acd0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110c60 con 0x7f3884105af0 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f388acd0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3884110f60 con 0x7f38841016d0 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47870/0 (socket says 172.21.15.78:47870) 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 -- 172.21.15.78:0/3164788257 learned_addr learned my addr 172.21.15.78:0/3164788257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f388926d640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 0x7f388419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3888a6c640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 0x7f3884198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 -- 172.21.15.78:0/3164788257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 msgr2=0x7f3884198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 0x7f3884198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 -- 172.21.15.78:0/3164788257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 msgr2=0x7f388419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 0x7f388419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 -- 172.21.15.78:0/3164788257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388419d2f0 con 0x7f3884102050 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3888a6c640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 0x7f3884198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f388926d640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 0x7f388419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.028+0000 7f3883fff640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3870006010 tx=0x7f38700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.029+0000 7f3881ffb640 1 -- 172.21.15.78:0/3164788257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387003f030 con 0x7f3884102050 2024-09-16T01:11:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.029+0000 7f3881ffb640 1 -- 172.21.15.78:0/3164788257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3870003ce0 con 0x7f3884102050 2024-09-16T01:11:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.029+0000 7f3881ffb640 1 -- 172.21.15.78:0/3164788257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870035070 con 0x7f3884102050 2024-09-16T01:11:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.029+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388419d580 con 0x7f3884102050 2024-09-16T01:11:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.029+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841a4e70 con 0x7f3884102050 2024-09-16T01:11:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 Processor -- start 2024-09-16T01:11:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 -- start start 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d9410 0x5620eb2e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5620eb23d2f0 con 0x5620eb2e3cb0 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5620eb1e5270 con 0x5620eb2d8af0 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5620eb2392b0 con 0x5620eb2d9410 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f3881ffb640 1 -- 172.21.15.78:0/3164788257 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f3870042030 con 0x7f3884102050 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388926d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.032+0000 7f388926d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41292/0 (socket says 172.21.15.78:41292) 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 -- 172.21.15.78:0/897977604 learned_addr learned my addr 172.21.15.78:0/897977604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f3888a6c640 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d9410 0x5620eb2e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 -- 172.21.15.78:0/897977604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d9410 msgr2=0x5620eb2e3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d9410 0x5620eb2e3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 -- 172.21.15.78:0/897977604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2e3cb0 msgr2=0x5620eb2e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 -- 172.21.15.78:0/897977604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620eb1e4340 con 0x5620eb2d8af0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f388926d640 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f387c009fb0 tx=0x7f387c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c3fdd3318a91315 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f3881ffb640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 0x7f38740a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f386effd640 1 -- 172.21.15.78:0/897977604 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c016070 con 0x5620eb2d8af0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f3881ffb640 1 -- 172.21.15.78:0/3164788257 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f38700f09e0 con 0x7f3884102050 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f386effd640 1 -- 172.21.15.78:0/897977604 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c012bf0 con 0x5620eb2d8af0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f386effd640 1 -- 172.21.15.78:0/897977604 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c01bcc0 con 0x5620eb2d8af0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.033+0000 7f3888a6c640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 0x7f38740a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 -- 172.21.15.78:0/897977604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 msgr2=0x5620eb2d8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f387c009fb0 tx=0x7f387c010800 comp rx=0 tx=0).stop 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f3888a6c640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 0x7f38740a5b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x5620eb2d5ec0 tx=0x7f3878009450 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 -- 172.21.15.78:0/897977604 shutdown_connections 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d9410 0x5620eb2e3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 --2- 172.21.15.78:0/897977604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2d8ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 -- 172.21.15.78:0/897977604 >> 172.21.15.78:0/897977604 conn(0x5620eb2d5060 msgr2=0x5620eb2d5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 -- 172.21.15.78:0/897977604 shutdown_connections 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.034+0000 7f388da6a740 1 -- 172.21.15.78:0/897977604 wait complete. 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 -- start start 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 Processor -- start 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d8af0 0x5620eb2e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5620eb2e6880 con 0x5620eb2d9410 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5620eb1c6290 con 0x5620eb2e3cb0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.035+0000 7f388da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5620eb2e6a00 con 0x5620eb2d8af0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f388da6a740 10 client.0 _ll_register_callbacks cb 0x5620eb1f7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47882/0 (socket says 172.21.15.78:47882) 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 -- 172.21.15.78:0/1132857740 learned_addr learned my addr 172.21.15.78:0/1132857740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f388926d640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3888a6c640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d8af0 0x5620eb2e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 -- 172.21.15.78:0/1132857740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d8af0 msgr2=0x5620eb2e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d8af0 0x5620eb2e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 -- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2e3cb0 msgr2=0x5620eb2e1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 -- 172.21.15.78:0/1132857740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620eaf50210 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3883fff640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38700edc50 tx=0x7f3870005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a33a4abe032f06e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3863fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870049070 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f38700c5190 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387c012bf0 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f38700b8a50 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f388da6a740 10 client.15396 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.036+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5620eb01c620 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c012bf0 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f38700bbc00 con 0x5620eb2d9410 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 10 client.15396 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 20 client.15396 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb2f3bc0 0x5620eb2f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 4 client.15396 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620eb24aa20 con 0x5620eb2f3bc0 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.037+0000 7f3888a6c640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb2f3bc0 0x5620eb2f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.038+0000 7f3863fff640 10 client.15396 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.038+0000 7f3888a6c640 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb2f3bc0 0x5620eb2f5fb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2d88b1221c5c18e server_cookie=22e977eb8fdad3af in_seq=0 out_seq=0 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f3863fff640 1 -- 172.21.15.78:0/1132857740 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5620eb24aa20 con 0x5620eb2f3bc0 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f3863fff640 10 client.15396 handle_command_reply: tid=0 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 client.15396 shutdown 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 msgr2=0x5620eb2e1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38700edc50 tx=0x7f3870005bd0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 shutdown_connections 2024-09-16T01:11:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb2f3bc0 0x5620eb2f5fb0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2e3cb0 0x5620eb2e1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2e1100 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f3863fff640 0 client.15396 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 --2- 172.21.15.78:0/1132857740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2d8af0 0x5620eb2e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 >> 172.21.15.78:0/1132857740 conn(0x5620eb2d5060 msgr2=0x5620eb2d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 shutdown_connections 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 1 -- 172.21.15.78:0/1132857740 wait complete. 2024-09-16T01:11:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.039+0000 7f388da6a740 20 client.15396 trim_cache size 0 max 16384 2024-09-16T01:11:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.054+0000 7f388da6a740 1 Processor -- start 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.054+0000 7f388da6a740 1 -- start start 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2e1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2fbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f38841b17a0 con 0x5620eb2d9410 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f38841af070 con 0x5620eb2d8af0 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38841acea0 con 0x5620eb2e3cb0 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47888/0 (socket says 172.21.15.78:47888) 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 -- 172.21.15.78:0/1837839217 learned_addr learned my addr 172.21.15.78:0/1837839217 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f388926d640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2fbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3888a6c640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2e1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 -- 172.21.15.78:0/1837839217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 msgr2=0x5620eb2fbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2fbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 -- 172.21.15.78:0/1837839217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 msgr2=0x5620eb2e1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2e1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 -- 172.21.15.78:0/1837839217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620eb2f6630 con 0x5620eb2d9410 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3888a6c640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2e1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.055+0000 7f3883fff640 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3870004650 tx=0x7f38700f0c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660cfb75a46f8163 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f386e7fc640 1 -- 172.21.15.78:0/1837839217 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870049070 con 0x5620eb2d9410 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f386e7fc640 1 -- 172.21.15.78:0/1837839217 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38700bceb0 con 0x5620eb2d9410 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f386e7fc640 1 -- 172.21.15.78:0/1837839217 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387003bcb0 con 0x5620eb2d9410 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 -- 172.21.15.78:0/1837839217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 msgr2=0x5620eb2fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3870004650 tx=0x7f38700f0c60 comp rx=0 tx=0).stop 2024-09-16T01:11:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 -- 172.21.15.78:0/1837839217 shutdown_connections 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2fbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d9410 0x5620eb2fb8c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.056+0000 7f388da6a740 1 --2- 172.21.15.78:0/1837839217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d8af0 0x5620eb2e1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- 172.21.15.78:0/1837839217 >> 172.21.15.78:0/1837839217 conn(0x5620eb2d5060 msgr2=0x5620eb2e2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- 172.21.15.78:0/1837839217 shutdown_connections 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- 172.21.15.78:0/1837839217 wait complete. 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- start start 2024-09-16T01:11:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 Processor -- start 2024-09-16T01:11:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 0x5620eb2de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ad530 con 0x5620eb2d8af0 2024-09-16T01:11:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ae9a0 con 0x5620eb2d9410 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38841adc00 con 0x5620eb2e3cb0 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.057+0000 7f388da6a740 10 client.0 _ll_register_callbacks cb 0x7f38700f6220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3888a6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3888a6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47896/0 (socket says 172.21.15.78:47896) 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3888a6c640 1 -- 172.21.15.78:0/4039595850 learned_addr learned my addr 172.21.15.78:0/4039595850 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 0x5620eb2de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f388926d640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 -- 172.21.15.78:0/4039595850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 msgr2=0x5620eb2de780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2de780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 -- 172.21.15.78:0/4039595850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 msgr2=0x5620eb2ddd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 -- 172.21.15.78:0/4039595850 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620eb2d02d0 con 0x5620eb2d9410 2024-09-16T01:11:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3888a6c640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3883fff640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 0x5620eb2de240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x5620eb2e33c0 tx=0x7f3870003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6370f3c88b67bfb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3863fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870049050 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f38700bdcf0 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.058+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38700bceb0 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f38700bc110 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 10 client.24920 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5620eb2d02d0 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38700bceb0 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3870031ad0 con 0x5620eb2d9410 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 10 client.24920 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 10 client.24920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 20 client.24920 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb35ed50 0x5620eb361170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 4 client.24920 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:11:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620eb244d70 con 0x5620eb35ed50 2024-09-16T01:11:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.059+0000 7f3888a6c640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb35ed50 0x5620eb361170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.060+0000 7f3863fff640 10 client.24920 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.060+0000 7f3888a6c640 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb35ed50 0x5620eb361170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d20d9059ab3d5d server_cookie=78ea4054d2a3dbd8 in_seq=0 out_seq=0 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.075+0000 7f3863fff640 1 -- 172.21.15.78:0/4039595850 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620eb244d70 con 0x5620eb35ed50 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.075+0000 7f3863fff640 10 client.24920 handle_command_reply: tid=0 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.075+0000 7f388da6a740 1 client.24920 shutdown 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 msgr2=0x5620eb2de240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 0x5620eb2de240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x5620eb2e33c0 tx=0x7f3870003a30 comp rx=0 tx=0).stop 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 shutdown_connections 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5620eb35ed50 0x5620eb361170 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5620eb2e3cb0 0x5620eb2de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f3863fff640 0 client.24920 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620eb2d9410 0x5620eb2de240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 --2- 172.21.15.78:0/4039595850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620eb2d8af0 0x5620eb2ddd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 >> 172.21.15.78:0/4039595850 conn(0x5620eb2d5060 msgr2=0x5620eb2d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 shutdown_connections 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 1 -- 172.21.15.78:0/4039595850 wait complete. 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388da6a740 20 client.24920 trim_cache size 0 max 16384 2024-09-16T01:11:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:11:04.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:11:04.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:11:04.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:11:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 msgr2=0x7f38740a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 0x7f38740a5b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x5620eb2d5ec0 tx=0x7f3878009450 comp rx=0 tx=0).stop 2024-09-16T01:11:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 msgr2=0x7f3884198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3870006010 tx=0x7f38700042f0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 shutdown_connections 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f38740a3690 0x7f38740a5b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3884105af0 0x7f388419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3884102050 0x7f3884198840 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 --2- 172.21.15.78:0/3164788257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38841016d0 0x7f3884198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.076+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 >> 172.21.15.78:0/3164788257 conn(0x7f3884075ec0 msgr2=0x7f38840ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.077+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 shutdown_connections 2024-09-16T01:11:04.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:04.077+0000 7f388acd0640 1 -- 172.21.15.78:0/3164788257 wait complete. 2024-09-16T01:11:04.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:04 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/267046000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:04.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/267046000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:04.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:04 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/267046000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:05.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-09-16T01:11:05.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:05 smithi078 ceph-mon[30937]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-09-16T01:11:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:05 smithi148 ceph-mon[35030]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-09-16T01:11:05.393 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-09-16T01:11:05.396 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-09-16T01:11:05.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-16T01:11:05.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-16T01:08:29.385380+0000', 'modified': '2024-09-16T01:10:45.934449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24485, 'mds_1': 24460, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1399158659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1399158659}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1399158659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/1850679548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 1850679548}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 1850679548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/1637418104', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1637418104}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1637418104}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.148:6837/330388800', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 330388800}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 330388800}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.193:6839/1136630649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 1136630649}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 1136630649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6839/3106047695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 3106047695}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 3106047695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24485, 24460, 24455]} 2024-09-16T01:11:05.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-16T01:11:05.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-16T01:11:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:05 smithi193 ceph-mon[35701]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/: 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 data 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data: 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:05.571 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:05.572 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:11:05.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-09-16T01:11:05.657 INFO:teuthology.orchestra.run.smithi078.stdout:[Service] 2024-09-16T01:11:05.657 INFO:teuthology.orchestra.run.smithi078.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-09-16T01:11:05.657 INFO:teuthology.orchestra.run.smithi078.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-09-16T01:11:05.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-09-16T01:11:05.792 INFO:teuthology.orchestra.run.smithi078.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-09-16T01:11:05.792 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 -- 172.21.15.78:0/3846381651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4481027c0 msgr2=0x7fc448106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 --2- 172.21.15.78:0/3846381651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4481027c0 0x7fc448106810 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc430009900 tx=0x7fc43002a410 comp rx=0 tx=0).stop 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 -- 172.21.15.78:0/3846381651 shutdown_connections 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 --2- 172.21.15.78:0/3846381651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc44810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 --2- 172.21.15.78:0/3846381651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4481027c0 0x7fc448106810 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 --2- 172.21.15.78:0/3846381651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc448101e10 0x7fc4481021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 -- 172.21.15.78:0/3846381651 >> 172.21.15.78:0/3846381651 conn(0x7fc4480fd8d0 msgr2=0x7fc4480ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 -- 172.21.15.78:0/3846381651 shutdown_connections 2024-09-16T01:11:06.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.370+0000 7fc44d050640 1 -- 172.21.15.78:0/3846381651 wait complete. 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 Processor -- start 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 -- start start 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4481027c0 0x7fc448074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc448074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc448111fa0 con 0x7fc448101e10 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc448111e20 con 0x7fc448106f40 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.371+0000 7fc44d050640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc448112120 con 0x7fc4481027c0 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57780/0 (socket says 172.21.15.78:57780) 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 -- 172.21.15.78:0/2909494644 learned_addr learned my addr 172.21.15.78:0/2909494644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc44759e640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc448074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc43e59c640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4481027c0 0x7fc448074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 -- 172.21.15.78:0/2909494644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4481027c0 msgr2=0x7fc448074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4481027c0 0x7fc448074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 -- 172.21.15.78:0/2909494644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 msgr2=0x7fc448074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc448074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 -- 172.21.15.78:0/2909494644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4481a6940 con 0x7fc448101e10 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc44759e640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc448074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc446d9d640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d950 tx=0x7fc43800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc444d99640 1 -- 172.21.15.78:0/2909494644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438017070 con 0x7fc448101e10 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4481a6c30 con 0x7fc448101e10 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc444d99640 1 -- 172.21.15.78:0/2909494644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc438009c30 con 0x7fc448101e10 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc444d99640 1 -- 172.21.15.78:0/2909494644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800f6b0 con 0x7fc448101e10 2024-09-16T01:11:06.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.372+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4481a7140 con 0x7fc448101e10 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 Processor -- start 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 -- start start 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc444d99640 1 -- 172.21.15.78:0/2909494644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7fc43800f850 con 0x7fc448101e10 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67e2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67effa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5581c6749090 con 0x5581c67e32f0 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581c66f0a70 con 0x5581c67e29d0 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc44f429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581c6745050 con 0x5581c67edbb0 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.375+0000 7fc446d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc446d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57792/0 (socket says 172.21.15.78:57792) 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67e2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc446d9d640 1 -- 172.21.15.78:0/4076736479 learned_addr learned my addr 172.21.15.78:0/4076736479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc43e59c640 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 -- 172.21.15.78:0/4076736479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 msgr2=0x5581c67effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67effa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 -- 172.21.15.78:0/4076736479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 msgr2=0x5581c67ed670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 -- 172.21.15.78:0/4076736479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c66fd130 con 0x5581c67e29d0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc446d9d640 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc44759e640 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67e2db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc440005fd0 tx=0x7fc440011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cfda472ac78c1a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc43d59a640 1 -- 172.21.15.78:0/4076736479 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440004d20 con 0x5581c67e29d0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc43d59a640 1 -- 172.21.15.78:0/4076736479 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44000f9a0 con 0x5581c67e29d0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc43d59a640 1 -- 172.21.15.78:0/4076736479 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4400045b0 con 0x5581c67e29d0 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.376+0000 7fc444d99640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 0x7fc4340a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc446d9d640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 0x7fc4340a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc444d99640 1 -- 172.21.15.78:0/2909494644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc4380cff30 con 0x7fc448101e10 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 -- 172.21.15.78:0/4076736479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 msgr2=0x5581c67e2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc446d9d640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 0x7fc4340a5a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc43801d000 tx=0x7fc4380d3a00 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67e2db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc440005fd0 tx=0x7fc440011bf0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 -- 172.21.15.78:0/4076736479 shutdown_connections 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 --2- 172.21.15.78:0/4076736479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67e2db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 -- 172.21.15.78:0/4076736479 >> 172.21.15.78:0/4076736479 conn(0x5581c67def60 msgr2=0x5581c67df340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 -- 172.21.15.78:0/4076736479 shutdown_connections 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.377+0000 7fc44f429740 1 -- 172.21.15.78:0/4076736479 wait complete. 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 -- start start 2024-09-16T01:11:06.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 Processor -- start 2024-09-16T01:11:06.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67ed440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5581c67f0780 con 0x5581c67e32f0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581c6703f20 con 0x5581c67e29d0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581c67f0900 con 0x5581c67edbb0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44f429740 10 client.0 _ll_register_callbacks cb 0x5581c66cd030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57796/0 (socket says 172.21.15.78:57796) 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 -- 172.21.15.78:0/1447828612 learned_addr learned my addr 172.21.15.78:0/1447828612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc44759e640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67ed440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 -- 172.21.15.78:0/1447828612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 msgr2=0x5581c67eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 -- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 msgr2=0x5581c67ed440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67ed440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.378+0000 7fc43e59c640 1 -- 172.21.15.78:0/1447828612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c64ba2b0 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc43e59c640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc43002a930 tx=0x7fc430031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e9d96ac1dd0c95 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430035040 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc430004240 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44000f9a0 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc430004440 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc44f429740 10 client.15414 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5581c67fc090 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44000f9a0 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.379+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc43003c6a0 con 0x5581c67e32f0 2024-09-16T01:11:06.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 10 client.15414 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 10 client.15414 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 20 client.15414 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c67fe700 0x5581c67ffb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 4 client.15414 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581c6756600 con 0x5581c67fe700 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44759e640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c67fe700 0x5581c67ffb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc44759e640 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c67fe700 0x5581c67ffb70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a71049f1a0a6863 server_cookie=71d20a2d61edf6a6 in_seq=0 out_seq=0 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.380+0000 7fc42e7fc640 10 client.15414 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc42e7fc640 1 -- 172.21.15.78:0/1447828612 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581c6756600 con 0x5581c67fe700 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc42e7fc640 10 client.15414 handle_command_reply: tid=0 2024-09-16T01:11:06.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 client.15414 shutdown 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 msgr2=0x5581c67ea790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc43002a930 tx=0x7fc430031850 comp rx=0 tx=0).stop 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 shutdown_connections 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c67fe700 0x5581c67ffb70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c67eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e32f0 0x5581c67ea790 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 --2- 172.21.15.78:0/1447828612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e29d0 0x5581c67ed440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 >> 172.21.15.78:0/1447828612 conn(0x5581c67def60 msgr2=0x5581c67e4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc42e7fc640 0 client.15414 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 shutdown_connections 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 1 -- 172.21.15.78:0/1447828612 wait complete. 2024-09-16T01:11:06.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.381+0000 7fc44f429740 20 client.15414 trim_cache size 0 max 16384 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.399+0000 7fc44f429740 1 Processor -- start 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.399+0000 7fc44f429740 1 -- start start 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e32f0 0x5581c6807e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c6808390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5581c67fdeb0 con 0x5581c67e29d0 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581c67fe580 con 0x5581c67e32f0 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44f429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581c67fec50 con 0x5581c67edbb0 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57812/0 (socket says 172.21.15.78:57812) 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 -- 172.21.15.78:0/283821689 learned_addr learned my addr 172.21.15.78:0/283821689 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc43e59c640 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e32f0 0x5581c6807e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 -- 172.21.15.78:0/283821689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 msgr2=0x5581c6808390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c6808390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 -- 172.21.15.78:0/283821689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e32f0 msgr2=0x5581c6807e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e32f0 0x5581c6807e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 -- 172.21.15.78:0/283821689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c67ded90 con 0x5581c67e29d0 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.400+0000 7fc44759e640 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc440004160 tx=0x7fc440009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435491683cdb8bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.401+0000 7fc43cd99640 1 -- 172.21.15.78:0/283821689 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44001e040 con 0x5581c67e29d0 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.401+0000 7fc43cd99640 1 -- 172.21.15.78:0/283821689 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44002b030 con 0x5581c67e29d0 2024-09-16T01:11:06.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.401+0000 7fc43cd99640 1 -- 172.21.15.78:0/283821689 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4400266a0 con 0x5581c67e29d0 2024-09-16T01:11:06.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- 172.21.15.78:0/283821689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 msgr2=0x5581c67fedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc440004160 tx=0x7fc440009f20 comp rx=0 tx=0).stop 2024-09-16T01:11:06.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- 172.21.15.78:0/283821689 shutdown_connections 2024-09-16T01:11:06.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67edbb0 0x5581c6808390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67e32f0 0x5581c6807e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 --2- 172.21.15.78:0/283821689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c67fedd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- 172.21.15.78:0/283821689 >> 172.21.15.78:0/283821689 conn(0x5581c67def60 msgr2=0x5581c67e9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- 172.21.15.78:0/283821689 shutdown_connections 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- 172.21.15.78:0/283821689 wait complete. 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 -- start start 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.402+0000 7fc44f429740 1 Processor -- start 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67e32f0 0x5581c68037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67edbb0 0x5581c686f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5581c67ff9f0 con 0x5581c67e29d0 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581c67ff320 con 0x5581c67edbb0 2024-09-16T01:11:06.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581c68000c0 con 0x5581c67e32f0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44f429740 10 client.0 _ll_register_callbacks cb 0x7fc430009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57824/0 (socket says 172.21.15.78:57824) 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 -- 172.21.15.78:0/15415072 learned_addr learned my addr 172.21.15.78:0/15415072 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc43e59c640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67e32f0 0x5581c68037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67edbb0 0x5581c686f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 -- 172.21.15.78:0/15415072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67e32f0 msgr2=0x5581c68037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67e32f0 0x5581c68037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 -- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67edbb0 msgr2=0x5581c686f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67edbb0 0x5581c686f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 -- 172.21.15.78:0/15415072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c686fb50 con 0x5581c67e29d0 2024-09-16T01:11:06.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc44759e640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc440009810 tx=0x7fc440005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1ff9a9e800daba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.403+0000 7fc42e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44001e040 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc440003a30 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44002b030 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc440005340 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 10 client.15420 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5581c6870e00 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44002b030 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc440018070 con 0x5581c67e29d0 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 10 client.15420 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 10 client.15420 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 20 client.15420 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:06.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c6875280 0x5581c6873f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 4 client.15420 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.404+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 --> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] -- command(tid 0: {"prefix": "status"}) -- 0x5581c6750b10 con 0x5581c6875280 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc43e59c640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c6875280 0x5581c6873f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc43e59c640 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c6875280 0x5581c6873f90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e86bfd39f1cfa14f server_cookie=7833be4eb9593361 in_seq=0 out_seq=0 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc42e7fc640 10 client.15420 ms_handle_connect on v2:172.21.15.148:6836/330388800 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc42e7fc640 1 -- 172.21.15.78:0/15415072 <== mds.2 v2:172.21.15.148:6836/330388800 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5581c6750b10 con 0x5581c6875280 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc42e7fc640 10 client.15420 handle_command_reply: tid=0 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc44f429740 1 client.15420 shutdown 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 msgr2=0x5581c6803290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc440009810 tx=0x7fc440005c30 comp rx=0 tx=0).stop 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 shutdown_connections 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.405+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:6836/330388800,v1:172.21.15.148:6837/330388800] conn(0x5581c6875280 0x5581c6873f90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581c67edbb0 0x5581c686f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581c67e32f0 0x5581c68037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 --2- 172.21.15.78:0/15415072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581c67e29d0 0x5581c6803290 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 >> 172.21.15.78:0/15415072 conn(0x5581c67def60 msgr2=0x5581c67e9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc42e7fc640 0 client.15420 ms_handle_reset on v2:172.21.15.148:6836/330388800 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 shutdown_connections 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 1 -- 172.21.15.78:0/15415072 wait complete. 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44f429740 20 client.15420 trim_cache size 0 max 16384 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:11:06.423 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24455, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 121.311847778, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 25, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 94, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 93, 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 179.718860116 2024-09-16T01:11:06.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 msgr2=0x7fc4340a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 0x7fc4340a5a20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc43801d000 tx=0x7fc4380d3a00 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 msgr2=0x7fc4480744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc43800d950 tx=0x7fc43800de10 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 shutdown_connections 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc4340a3560 0x7fc4340a5a20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc448106f40 0x7fc448074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4481027c0 0x7fc448074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 --2- 172.21.15.78:0/2909494644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc448101e10 0x7fc4480744f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.406+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 >> 172.21.15.78:0/2909494644 conn(0x7fc4480fd8d0 msgr2=0x7fc4480fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:06.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.407+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 shutdown_connections 2024-09-16T01:11:06.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:06.407+0000 7fc44d050640 1 -- 172.21.15.78:0/2909494644 wait complete. 2024-09-16T01:11:07.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-16T01:11:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:07 smithi078 ceph-mon[30937]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-09-16T01:11:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:07 smithi148 ceph-mon[35030]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-09-16T01:11:07.484 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:07 smithi193 ceph-mon[35701]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 31 op/s 2024-09-16T01:11:08.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 -- 172.21.15.78:0/2345853392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102ef0 msgr2=0x7f52e010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 --2- 172.21.15.78:0/2345853392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102ef0 0x7f52e010da40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009900 tx=0x7f52d802a480 comp rx=0 tx=0).stop 2024-09-16T01:11:08.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 -- 172.21.15.78:0/2345853392 shutdown_connections 2024-09-16T01:11:08.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 --2- 172.21.15.78:0/2345853392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102ef0 0x7f52e010da40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 --2- 172.21.15.78:0/2345853392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102570 0x7f52e01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 --2- 172.21.15.78:0/2345853392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 0x7f52e0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 -- 172.21.15.78:0/2345853392 >> 172.21.15.78:0/2345853392 conn(0x7f52e00fc7d0 msgr2=0x7f52e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 -- 172.21.15.78:0/2345853392 shutdown_connections 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.989+0000 7f52e4964640 1 -- 172.21.15.78:0/2345853392 wait complete. 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 Processor -- start 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 -- start start 2024-09-16T01:11:08.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 0x7f52e0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102ef0 0x7f52e019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f52e0110f80 con 0x7f52e0102570 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f52e0110e00 con 0x7f52e0102ef0 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52e4964640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f52e0111100 con 0x7f52e0100e20 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52ddd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52ddd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57842/0 (socket says 172.21.15.78:57842) 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52ddd9b640 1 -- 172.21.15.78:0/1707752722 learned_addr learned my addr 172.21.15.78:0/1707752722 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52ded9d640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102ef0 0x7f52e019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.990+0000 7f52de59c640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 0x7f52e0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 -- 172.21.15.78:0/1707752722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 msgr2=0x7f52e0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 0x7f52e0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 -- 172.21.15.78:0/1707752722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102ef0 msgr2=0x7f52e019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102ef0 0x7f52e019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 -- 172.21.15.78:0/1707752722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e019d4b0 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f52c800e9f0 tx=0x7f52c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52d77fe640 1 -- 172.21.15.78:0/1707752722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c800cd80 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e019d7a0 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52d77fe640 1 -- 172.21.15.78:0/1707752722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52c8010910 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52d77fe640 1 -- 172.21.15.78:0/1707752722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8018990 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.991+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e01a5080 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.993+0000 7f52e6d3d740 1 Processor -- start 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.993+0000 7f52d77fe640 1 -- 172.21.15.78:0/1707752722 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f52c8018b30 con 0x7f52e0102570 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 -- start start 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3de2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3de4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3d3e090 con 0x55f6d3dd79d0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3ce5a70 con 0x55f6d3de2bb0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3d3a050 con 0x55f6d3dd82f0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57856/0 (socket says 172.21.15.78:57856) 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 -- 172.21.15.78:0/3670862974 learned_addr learned my addr 172.21.15.78:0/3670862974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3de4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52de59c640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3de2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 -- 172.21.15.78:0/3670862974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 msgr2=0x55f6d3de2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3de2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 -- 172.21.15.78:0/3670862974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 msgr2=0x55f6d3de4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3de4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ded9d640 1 -- 172.21.15.78:0/3670862974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d3cf2130 con 0x55f6d3dd79d0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3de4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.994+0000 7f52d77fe640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 0x7f52cc0a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52ded9d640 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f52d8033000 tx=0x7f52d8005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b9ad499f597034 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52de59c640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 0x7f52cc0a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52d77fe640 1 -- 172.21.15.78:0/1707752722 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f52c80d05d0 con 0x7f52e0102570 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52d4ff9640 1 -- 172.21.15.78:0/3670862974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8035030 con 0x55f6d3dd79d0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52d4ff9640 1 -- 172.21.15.78:0/3670862974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d8040070 con 0x55f6d3dd79d0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52d4ff9640 1 -- 172.21.15.78:0/3670862974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d80319a0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3670862974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 msgr2=0x55f6d3dd7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f52d8033000 tx=0x7f52d8005e70 comp rx=0 tx=0).stop 2024-09-16T01:11:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.995+0000 7f52de59c640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 0x7f52cc0a5a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55f6d3dd4dc0 tx=0x7f52d0009680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3670862974 shutdown_connections 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3de4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3de2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3670862974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dd7db0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3670862974 >> 172.21.15.78:0/3670862974 conn(0x55f6d3dd3f60 msgr2=0x55f6d3dd4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3670862974 shutdown_connections 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3670862974 wait complete. 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- start start 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 Processor -- start 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3de5780 con 0x55f6d3de2bb0 2024-09-16T01:11:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3cf8f20 con 0x55f6d3dd79d0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.996+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3de5900 con 0x55f6d3dd82f0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52e6d3d740 10 client.0 _ll_register_callbacks cb 0x55f6d3cc2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52ded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45998/0 (socket says 172.21.15.78:45998) 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 -- 172.21.15.78:0/4158112274 learned_addr learned my addr 172.21.15.78:0/4158112274 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 -- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 msgr2=0x55f6d3ddf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 -- 172.21.15.78:0/4158112274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3de2bb0 msgr2=0x55f6d3ddfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 -- 172.21.15.78:0/4158112274 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d3aaf2b0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52de59c640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f52d0004260 tx=0x7f52d0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342927cda34462b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0011d60 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f52d00203f0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.997+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52d8040070 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f52d00205f0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 10 client.24950 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6d3df1150 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d8040070 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f52d000f040 con 0x55f6d3dd79d0 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 10 client.24950 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 10 client.24950 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 20 client.24950 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3df3770 0x55f6d3df4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 4 client.24950 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6d3d4b600 con 0x55f6d3df3770 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.998+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3df3770 0x55f6d3df4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.999+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3df3770 0x55f6d3df4be0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=591c204e26aae432 server_cookie=f258bde676d7a2da in_seq=0 out_seq=0 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.999+0000 7f52c5ffb640 10 client.24950 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.999+0000 7f52c5ffb640 1 -- 172.21.15.78:0/4158112274 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f6d3d4b600 con 0x55f6d3df3770 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:07.999+0000 7f52c5ffb640 10 client.24950 handle_command_reply: tid=0 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 client.24950 shutdown 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 msgr2=0x55f6d3de2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f52d0004260 tx=0x7f52d0004a40 comp rx=0 tx=0).stop 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 shutdown_connections 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3df3770 0x55f6d3df4be0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52c5ffb640 0 client.24950 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/4158112274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd79d0 0x55f6d3de2490 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 >> 172.21.15.78:0/4158112274 conn(0x55f6d3dd3f60 msgr2=0x55f6d3dd9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 shutdown_connections 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 1 -- 172.21.15.78:0/4158112274 wait complete. 2024-09-16T01:11:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.000+0000 7f52e6d3d740 20 client.24950 trim_cache size 0 max 16384 2024-09-16T01:11:08.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 Processor -- start 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 -- start start 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd82f0 0x55f6d3dcece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df2f20 con 0x55f6d3dd79d0 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df35f0 con 0x55f6d3dd82f0 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df3cc0 con 0x55f6d3de2bb0 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57882/0 (socket says 172.21.15.78:57882) 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 -- 172.21.15.78:0/3157469091 learned_addr learned my addr 172.21.15.78:0/3157469091 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:08.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd82f0 0x55f6d3dcece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52ded9d640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 -- 172.21.15.78:0/3157469091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 msgr2=0x55f6d3ddfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 -- 172.21.15.78:0/3157469091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd82f0 msgr2=0x55f6d3dcece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd82f0 0x55f6d3dcece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52de59c640 1 -- 172.21.15.78:0/3157469091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6d3dde7b0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.018+0000 7f52ded9d640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.019+0000 7f52de59c640 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52d0013f20 tx=0x7f52d0005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b4c9a4e7737cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.019+0000 7f52c7fff640 1 -- 172.21.15.78:0/3157469091 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0039070 con 0x55f6d3dd79d0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.019+0000 7f52c7fff640 1 -- 172.21.15.78:0/3157469091 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d00119b0 con 0x55f6d3dd79d0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.019+0000 7f52c7fff640 1 -- 172.21.15.78:0/3157469091 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0028910 con 0x55f6d3dd79d0 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3157469091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 msgr2=0x55f6d3df3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52d0013f20 tx=0x7f52d0005ac0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3157469091 shutdown_connections 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3dd82f0 0x55f6d3dcece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/3157469091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd79d0 0x55f6d3df3e40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3157469091 >> 172.21.15.78:0/3157469091 conn(0x55f6d3dd3f60 msgr2=0x55f6d3dd27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3157469091 shutdown_connections 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- 172.21.15.78:0/3157469091 wait complete. 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.020+0000 7f52e6d3d740 1 -- start start 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 Processor -- start 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dfd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df4a60 con 0x55f6d3dd82f0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df4390 con 0x55f6d3de2bb0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f6d3df5130 con 0x55f6d3dd79d0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52e6d3d740 10 client.0 _ll_register_callbacks cb 0x7f52d0003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ddd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ddd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57886/0 (socket says 172.21.15.78:57886) 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ddd9b640 1 -- 172.21.15.78:0/1200653221 learned_addr learned my addr 172.21.15.78:0/1200653221 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52de59c640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dfd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 -- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 msgr2=0x55f6d3dfd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dfd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 -- 172.21.15.78:0/1200653221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 msgr2=0x55f6d3ddc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ded9d640 1 -- 172.21.15.78:0/1200653221 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6d3ddcd70 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52ddd9b640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.021+0000 7f52de59c640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dfd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52ded9d640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddc830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52d8005c40 tx=0x7f52d8003f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16fb277f29d00295 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8035070 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f52d8005020 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52d00119b0 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f52d8009520 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52e6d3d740 10 client.24956 fetch_fsmap learned FSMap version 25 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6d3dfbaa0 con 0x55f6d3de2bb0 2024-09-16T01:11:08.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d00119b0 con 0x55f6d3de2bb0 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.022+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f52d80035f0 con 0x55f6d3de2bb0 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 10 client.24956 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 10 client.24956 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 20 client.24956 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3e62800 0x55f6d3e61510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 4 client.24956 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 --> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] -- command(tid 0: {"prefix": "status"}) -- 0x55f6d3d45b10 con 0x55f6d3e62800 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52de59c640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3e62800 0x55f6d3e61510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52de59c640 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3e62800 0x55f6d3e61510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=567513096696bc87 server_cookie=87a5dd9b965af822 in_seq=0 out_seq=0 2024-09-16T01:11:08.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.023+0000 7f52c5ffb640 10 client.24956 ms_handle_connect on v2:172.21.15.193:6838/1136630649 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52c5ffb640 1 -- 172.21.15.78:0/1200653221 <== mds.1 v2:172.21.15.193:6838/1136630649 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55f6d3d45b10 con 0x55f6d3e62800 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52c5ffb640 10 client.24956 handle_command_reply: tid=0 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 client.24956 shutdown 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 msgr2=0x55f6d3ddc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddc830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52d8005c40 tx=0x7f52d8003f40 comp rx=0 tx=0).stop 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 shutdown_connections 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:6838/1136630649,v1:172.21.15.193:6839/1136630649] conn(0x55f6d3e62800 0x55f6d3e61510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f6d3de2bb0 0x55f6d3ddc830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52c5ffb640 0 client.24956 ms_handle_reset on v2:172.21.15.193:6838/1136630649 2024-09-16T01:11:08.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6d3dd82f0 0x55f6d3ddc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 --2- 172.21.15.78:0/1200653221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f6d3dd79d0 0x55f6d3dfd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 >> 172.21.15.78:0/1200653221 conn(0x55f6d3dd3f60 msgr2=0x55f6d3dd9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 shutdown_connections 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 1 -- 172.21.15.78:0/1200653221 wait complete. 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.024+0000 7f52e6d3d740 20 client.24956 trim_cache size 0 max 16384 2024-09-16T01:11:08.037 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24460, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 123.938917589, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 25, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 94, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 93, 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 178.808279551 2024-09-16T01:11:08.038 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 msgr2=0x7f52cc0a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 0x7f52cc0a5a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55f6d3dd4dc0 tx=0x7f52d0009680 comp rx=0 tx=0).stop 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 msgr2=0x7f52e0198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f52c800e9f0 tx=0x7f52c800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 shutdown_connections 2024-09-16T01:11:08.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f52cc0a3560 0x7f52cc0a5a20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52e0102ef0 0x7f52e019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52e0102570 0x7f52e0198a40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 --2- 172.21.15.78:0/1707752722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52e0100e20 0x7f52e0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 >> 172.21.15.78:0/1707752722 conn(0x7f52e00fc7d0 msgr2=0x7f52e006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 shutdown_connections 2024-09-16T01:11:08.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:08.025+0000 7f52e4964640 1 -- 172.21.15.78:0/1707752722 wait complete. 2024-09-16T01:11:08.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-16T01:11:09.081 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:09.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:09 smithi078 ceph-mon[30937]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-09-16T01:11:09.356 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:09 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:09 smithi148 ceph-mon[35030]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-09-16T01:11:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:09 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:09 smithi193 ceph-mon[35701]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-09-16T01:11:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:09 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 -- 172.21.15.78:0/1766112271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390100e20 msgr2=0x7ff390101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 --2- 172.21.15.78:0/1766112271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390100e20 0x7ff390101200 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff384009900 tx=0x7ff38402a480 comp rx=0 tx=0).stop 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 -- 172.21.15.78:0/1766112271 shutdown_connections 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 --2- 172.21.15.78:0/1766112271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390102ef0 0x7ff39010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 --2- 172.21.15.78:0/1766112271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102570 0x7ff3901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 --2- 172.21.15.78:0/1766112271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390100e20 0x7ff390101200 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.584+0000 7ff3972fb640 1 -- 172.21.15.78:0/1766112271 >> 172.21.15.78:0/1766112271 conn(0x7ff3900fc7d0 msgr2=0x7ff3900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- 172.21.15.78:0/1766112271 shutdown_connections 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- 172.21.15.78:0/1766112271 wait complete. 2024-09-16T01:11:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 Processor -- start 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- start start 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390100e20 0x7ff3901993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102ef0 0x7ff3901a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff390110f80 con 0x7ff390102570 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff390110e00 con 0x7ff390102ef0 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.585+0000 7ff3972fb640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff390111100 con 0x7ff390100e20 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff395097640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390100e20 0x7ff3901993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57908/0 (socket says 172.21.15.78:57908) 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 -- 172.21.15.78:0/2684136161 learned_addr learned my addr 172.21.15.78:0/2684136161 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff395898640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102ef0 0x7ff3901a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 -- 172.21.15.78:0/2684136161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390100e20 msgr2=0x7ff3901993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390100e20 0x7ff3901993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 -- 172.21.15.78:0/2684136161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102ef0 msgr2=0x7ff3901a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102ef0 0x7ff3901a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 -- 172.21.15.78:0/2684136161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3901a4990 con 0x7ff390102570 2024-09-16T01:11:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff394896640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff38800ebc0 tx=0x7ff38800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff3827fc640 1 -- 172.21.15.78:0/2684136161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388010630 con 0x7ff390102570 2024-09-16T01:11:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3901a4b60 con 0x7ff390102570 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff3827fc640 1 -- 172.21.15.78:0/2684136161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff388010cb0 con 0x7ff390102570 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff3827fc640 1 -- 172.21.15.78:0/2684136161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388018bd0 con 0x7ff390102570 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.586+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3901a50a0 con 0x7ff390102570 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3827fc640 1 -- 172.21.15.78:0/2684136161 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7ff3880107d0 con 0x7ff390102570 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 Processor -- start 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 -- start start 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9381362f0 0x55e938140670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 0x55e938142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e93809c090 con 0x55e938140bb0 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e938043a70 con 0x55e9381359d0 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff3996d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e938098050 con 0x55e9381362f0 2024-09-16T01:11:09.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff394896640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 0x55e938142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff395898640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff395898640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:46042/0 (socket says 172.21.15.78:46042) 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.589+0000 7ff395898640 1 -- 172.21.15.78:0/3785062514 learned_addr learned my addr 172.21.15.78:0/3785062514 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395097640 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9381362f0 0x55e938140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff3827fc640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 0x7ff37c0a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 -- 172.21.15.78:0/3785062514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9381362f0 msgr2=0x55e938140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9381362f0 0x55e938140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 -- 172.21.15.78:0/3785062514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 msgr2=0x55e938142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 0x55e938142fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 -- 172.21.15.78:0/3785062514 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e938050130 con 0x55e9381359d0 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff394896640 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 0x55e938142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff3827fc640 1 -- 172.21.15.78:0/2684136161 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff3880d0a50 con 0x7ff390102570 2024-09-16T01:11:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 0x7ff37c0a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009d10 tx=0x7ff38c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1161fa90298a711b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff395898640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 0x7ff37c0a5a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff38c022000 tx=0x7ff38c01f000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff37b7fe640 1 -- 172.21.15.78:0/3785062514 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0140d0 con 0x55e9381359d0 2024-09-16T01:11:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff37b7fe640 1 -- 172.21.15.78:0/3785062514 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38c014270 con 0x55e9381359d0 2024-09-16T01:11:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.590+0000 7ff37b7fe640 1 -- 172.21.15.78:0/3785062514 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00f680 con 0x55e9381359d0 2024-09-16T01:11:09.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 -- 172.21.15.78:0/3785062514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 msgr2=0x55e938135db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009d10 tx=0x7ff38c0117b0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 -- 172.21.15.78:0/3785062514 shutdown_connections 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e938140bb0 0x55e938142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9381362f0 0x55e938140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 --2- 172.21.15.78:0/3785062514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938135db0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 -- 172.21.15.78:0/3785062514 >> 172.21.15.78:0/3785062514 conn(0x55e938131f60 msgr2=0x55e938132340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 -- 172.21.15.78:0/3785062514 shutdown_connections 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.591+0000 7ff3996d4740 1 -- 172.21.15.78:0/3785062514 wait complete. 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 -- start start 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 Processor -- start 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e93813d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e938143780 con 0x55e9381359d0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e938056f20 con 0x55e9381362f0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e938143900 con 0x55e938140bb0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff3996d4740 10 client.0 _ll_register_callbacks cb 0x55e938020030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff395097640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff395097640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57924/0 (socket says 172.21.15.78:57924) 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff395097640 1 -- 172.21.15.78:0/2903502262 learned_addr learned my addr 172.21.15.78:0/2903502262 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.592+0000 7ff395898640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 -- 172.21.15.78:0/2903502262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 msgr2=0x55e93813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93813dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 -- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 msgr2=0x55e93813d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff394896640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e93813d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e93813d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 -- 172.21.15.78:0/2903502262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e937e0d2b0 con 0x55e9381359d0 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395898640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff394896640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e93813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff395097640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff38402a9a0 tx=0x7ff384004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270191da9417a3f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff378ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384035070 con 0x55e9381359d0 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff38c014270 con 0x55e9381359d0 2024-09-16T01:11:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff38403e7d0 con 0x55e9381359d0 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff384051050 con 0x55e9381359d0 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff3996d4740 10 client.15450 fetch_fsmap learned FSMap version 26 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e93814f0f0 con 0x55e9381359d0 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.593+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38c014270 con 0x55e9381359d0 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7ff384031540 con 0x55e9381359d0 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 10 client.15450 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 10 client.15450 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 20 client.15450 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e938152b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 4 client.15450 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9380a9600 con 0x55e938151690 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff394896640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e938152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff378ff9640 10 client.15450 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.594+0000 7ff394896640 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e938152b00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4b687cda44785f server_cookie=aa1e221f182e75a6 in_seq=0 out_seq=0 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.595+0000 7ff378ff9640 1 -- 172.21.15.78:0/2903502262 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9380a9600 con 0x55e938151690 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.595+0000 7ff378ff9640 10 client.15450 handle_command_reply: tid=0 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.595+0000 7ff3996d4740 1 client.15450 shutdown 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 msgr2=0x55e938140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff38402a9a0 tx=0x7ff384004270 comp rx=0 tx=0).stop 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 shutdown_connections 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e938152b00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff378ff9640 0 client.15450 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e93813d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2903502262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938140460 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 >> 172.21.15.78:0/2903502262 conn(0x55e938131f60 msgr2=0x55e938137c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 shutdown_connections 2024-09-16T01:11:09.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 1 -- 172.21.15.78:0/2903502262 wait complete. 2024-09-16T01:11:09.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.596+0000 7ff3996d4740 20 client.15450 trim_cache size 0 max 16384 2024-09-16T01:11:09.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.613+0000 7ff3996d4740 1 Processor -- start 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.613+0000 7ff3996d4740 1 -- start start 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e938151d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93815ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e938150e40 con 0x55e9381359d0 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e938151510 con 0x55e9381362f0 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff3996d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e938151be0 con 0x55e938140bb0 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57930/0 (socket says 172.21.15.78:57930) 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 -- 172.21.15.78:0/1608724966 learned_addr learned my addr 172.21.15.78:0/1608724966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff394896640 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e938151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395898640 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93815ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 -- 172.21.15.78:0/1608724966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 msgr2=0x55e93815ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93815ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 -- 172.21.15.78:0/1608724966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 msgr2=0x55e938151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e938151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 -- 172.21.15.78:0/1608724966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e938136b70 con 0x55e9381359d0 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff395097640 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3840314e0 tx=0x7ff3840316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1283c9224a05201e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.614+0000 7ff37affd640 1 -- 172.21.15.78:0/1608724966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384035040 con 0x55e9381359d0 2024-09-16T01:11:09.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff37affd640 1 -- 172.21.15.78:0/1608724966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384005500 con 0x55e9381359d0 2024-09-16T01:11:09.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff37affd640 1 -- 172.21.15.78:0/1608724966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38403e7d0 con 0x55e9381359d0 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 -- 172.21.15.78:0/1608724966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 msgr2=0x55e938150fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3840314e0 tx=0x7ff3840316d0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 -- 172.21.15.78:0/1608724966 shutdown_connections 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e93815ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381362f0 0x55e938151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 --2- 172.21.15.78:0/1608724966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381359d0 0x55e938150fc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 -- 172.21.15.78:0/1608724966 >> 172.21.15.78:0/1608724966 conn(0x55e938131f60 msgr2=0x55e93813c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 -- 172.21.15.78:0/1608724966 shutdown_connections 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.615+0000 7ff3996d4740 1 -- 172.21.15.78:0/1608724966 wait complete. 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 -- start start 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 Processor -- start 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e938156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e938152980 con 0x55e9381362f0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9381522b0 con 0x55e9381359d0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e938153050 con 0x55e938140bb0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff3996d4740 10 client.0 _ll_register_callbacks cb 0x7ff38c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff394896640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff394896640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57932/0 (socket says 172.21.15.78:57932) 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff394896640 1 -- 172.21.15.78:0/2214842154 learned_addr learned my addr 172.21.15.78:0/2214842154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff395097640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.616+0000 7ff395898640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e938156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 -- 172.21.15.78:0/2214842154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 msgr2=0x55e938156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e938156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 -- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 msgr2=0x55e938155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 -- 172.21.15.78:0/2214842154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9381ba0b0 con 0x55e9381362f0 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff395898640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e938156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff394896640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff38809c460 tx=0x7ff3880d0d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb802df96a9e446 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff378ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388010e60 con 0x55e9381362f0 2024-09-16T01:11:09.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ff388098dd0 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff384005500 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff38809ce60 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff3996d4740 10 client.15456 fetch_fsmap learned FSMap version 26 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e9381bb310 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.617+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384005500 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7ff3880a4f40 con 0x55e9381362f0 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 10 client.15456 fetch_fsmap finished waiting for FSMap version 26 2024-09-16T01:11:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 10 client.15456 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 20 client.15456 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e9381bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 4 client.15456 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "status"}) -- 0x55e9380a3b10 con 0x55e938151690 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff395097640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e9381bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff378ff9640 10 client.15456 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.618+0000 7ff395097640 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e9381bec90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1cc322f62f28ef server_cookie=3a0a2a127a785fae in_seq=0 out_seq=0 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff378ff9640 1 -- 172.21.15.78:0/2214842154 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e9380a3b10 con 0x55e938151690 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff378ff9640 10 client.15456 handle_command_reply: tid=0 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff3996d4740 1 client.15456 shutdown 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 msgr2=0x55e938156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff38809c460 tx=0x7ff3880d0d80 comp rx=0 tx=0).stop 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 shutdown_connections 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.619+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e938151690 0x55e9381bec90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e938140bb0 0x55e938156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff378ff9640 0 client.15456 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9381362f0 0x55e938156300 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 --2- 172.21.15.78:0/2214842154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9381359d0 0x55e938155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 >> 172.21.15.78:0/2214842154 conn(0x55e938131f60 msgr2=0x55e938131000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 shutdown_connections 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 1 -- 172.21.15.78:0/2214842154 wait complete. 2024-09-16T01:11:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.620+0000 7ff3996d4740 20 client.15456 trim_cache size 0 max 16384 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "fa4731de-73c6-11ef-bceb-c7b262605968", 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 159.88832505799999, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 26, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 94, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 94, 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 171.84680236899999 2024-09-16T01:11:09.629 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 msgr2=0x7ff37c0a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 0x7ff37c0a5a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff38c022000 tx=0x7ff38c01f000 comp rx=0 tx=0).stop 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 msgr2=0x7ff390199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff38800ebc0 tx=0x7ff38800c6a0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 shutdown_connections 2024-09-16T01:11:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff37c0a3560 0x7ff37c0a5a20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff390102ef0 0x7ff3901a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff390102570 0x7ff390199910 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 --2- 172.21.15.78:0/2684136161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff390100e20 0x7ff3901993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 >> 172.21.15.78:0/2684136161 conn(0x7ff3900fc7d0 msgr2=0x7ff3900fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 shutdown_connections 2024-09-16T01:11:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:09.621+0000 7ff3972fb640 1 -- 172.21.15.78:0/2684136161 wait complete. 2024-09-16T01:11:09.632 INFO:teuthology.orchestra.run.smithi078.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-09-16T01:11:09.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-16T01:11:09.677 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/: 2024-09-16T01:11:09.677 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.677 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 data 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data: 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:total 35 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 . 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 base 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Sep 16 01:11 global 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 log 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_commit_ts 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_dynshmem 2024-09-16T01:11:09.678 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Sep 16 01:11 pg_hba.conf 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Sep 16 01:11 pg_ident.conf 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 pg_logical 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 pg_multixact 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_notify 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_replslot 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_serial 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_snapshots 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_stat 2024-09-16T01:11:09.679 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_stat_tmp 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 pg_subtrans 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_tblspc 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_twophase 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 pg_wal 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 pg_xact 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Sep 16 01:11 postgresql.auto.conf 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28074 Sep 16 01:11 postgresql.conf 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base: 2024-09-16T01:11:09.680 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 . 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 1 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 13493 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 13494 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/1: 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:total 7777 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 . 2024-09-16T01:11:09.681 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 .. 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:11:09.682 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:11 1259 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:11:09.683 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:11:09.684 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:11:09.685 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:11:09.686 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:11:09.687 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:11:09.688 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:11:09.689 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:11:09.690 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 16 01:11 2619 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2619_fsm 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:11:09.691 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:11:09.692 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:11:09.693 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:11:09.694 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:11:09.695 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2696 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:11:09.696 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:11:09.697 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:11:09.698 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:11:09.699 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:11:09.700 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:11:09.701 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:11:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:11:09.703 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:11:09.704 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:11:09.705 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:11:09.706 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:11:09.706 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:11:09.706 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:11:09.709 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:11:09.710 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-09-16T01:11:09.711 INFO:teuthology.orchestra.run.smithi078.stdout:total 7777 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 . 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 .. 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:11:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:11 1259 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:11:09.713 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:11:09.714 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:11:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:11:09.716 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:11:09.717 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:11:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:11:09.719 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 16 01:11 2619 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2619_fsm 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:11:09.720 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:11:09.721 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:11:09.722 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:11:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2696 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:11:09.724 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:11:09.725 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:11:09.726 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:11:09.727 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:11:09.728 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:11:09.729 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:11:09.730 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:11:09.731 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:11:09.732 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:11:09.733 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:11:09.734 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:11:09.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:11:09.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:11:09.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:11:09.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.739 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:total 7777 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 . 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 .. 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:11:09.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:11 1259 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:11:09.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:11:09.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:11:09.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:11:09.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:11:09.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:11:09.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:11:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 16 01:11 2619 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2619_fsm 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:11:09.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:11:09.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:11:09.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2696 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:11:09.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:11:09.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:11:09.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:11:09.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:11:09.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:11:09.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:11:09.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:11:09.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:11:09.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:11:09.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:11:09.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:11:09.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:11:09.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:11:09.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:11:09.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:11:09.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:11:09.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:11:09.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:11:09.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:11:09.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/global: 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:total 537 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Sep 16 01:11 . 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1213 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1213_fsm 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1213_vm 2024-09-16T01:11:09.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1214 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1214_fsm 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1214_vm 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 1232 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 1233 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1260 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1260_fsm 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1260_vm 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1261 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1261_fsm 2024-09-16T01:11:09.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1261_vm 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1262 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1262_fsm 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1262_vm 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2396 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2396_fsm 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2396_vm 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2397 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2671 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2672 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2676 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2677 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2694 2024-09-16T01:11:09.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2695 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2697 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2698 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2846 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2847 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2964 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2965 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2966 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2967 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3592 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3593 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4060 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4061 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4175 2024-09-16T01:11:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4176 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4177 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4178 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4181 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4182 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4183 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4184 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4185 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4186 2024-09-16T01:11:09.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6000 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6001 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6002 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6100 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6114 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6115 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 pg_control 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/log: 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.772 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.773 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:total 1 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 . 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 mappings 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Sep 16 01:11 replorigin_checkpoint 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 snapshots 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 .. 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 .. 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 . 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 members 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 offsets 2024-09-16T01:11:09.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 .. 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 0000 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 .. 2024-09-16T01:11:09.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 0000 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.777 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-09-16T01:11:09.778 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:11:09.779 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 0000 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.780 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:total 16384 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 . 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:11 000000010000000000000001 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 archive_status 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 .. 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:11:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:11:09.783 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 16 01:11 .. 2024-09-16T01:11:09.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 0000 2024-09-16T01:11:09.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-09-16T01:11:09.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-09-16T01:11:09.974 INFO:teuthology.orchestra.run.smithi078.stderr:dropping old tables... 2024-09-16T01:11:09.975 INFO:teuthology.orchestra.run.smithi078.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-09-16T01:11:09.976 INFO:teuthology.orchestra.run.smithi078.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-09-16T01:11:09.976 INFO:teuthology.orchestra.run.smithi078.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-09-16T01:11:09.976 INFO:teuthology.orchestra.run.smithi078.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-09-16T01:11:09.976 INFO:teuthology.orchestra.run.smithi078.stderr:creating tables... 2024-09-16T01:11:10.014 INFO:teuthology.orchestra.run.smithi078.stderr:generating data (client-side)... 2024-09-16T01:11:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.06 s, remaining 1.79 s) 2024-09-16T01:11:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.20 s, remaining 3.05 s) 2024-09-16T01:11:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.29 s, remaining 2.84 s) 2024-09-16T01:11:10.462 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-16T01:11:10.501 INFO:teuthology.orchestra.run.smithi078.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.45 s, remaining 3.16 s) 2024-09-16T01:11:10.680 INFO:teuthology.orchestra.run.smithi078.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.63 s, remaining 3.40 s) 2024-09-16T01:11:10.838 INFO:teuthology.orchestra.run.smithi078.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.79 s, remaining 3.41 s) 2024-09-16T01:11:10.925 INFO:teuthology.orchestra.run.smithi078.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.88 s, remaining 3.13 s) 2024-09-16T01:11:11.088 INFO:teuthology.orchestra.run.smithi078.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.04 s, remaining 3.11 s) 2024-09-16T01:11:11.237 INFO:teuthology.orchestra.run.smithi078.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.19 s, remaining 3.03 s) 2024-09-16T01:11:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.27 s, remaining 2.78 s) 2024-09-16T01:11:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:11 smithi148 ceph-mon[35030]: pgmap v14: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 171 KiB/s wr, 33 op/s 2024-09-16T01:11:11.454 INFO:teuthology.orchestra.run.smithi078.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.40 s, remaining 2.68 s) 2024-09-16T01:11:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:11 smithi078 ceph-mon[30937]: pgmap v14: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 171 KiB/s wr, 33 op/s 2024-09-16T01:11:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.50 s, remaining 2.49 s) 2024-09-16T01:11:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:11 smithi193 ceph-mon[35701]: pgmap v14: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 171 KiB/s wr, 33 op/s 2024-09-16T01:11:11.666 INFO:teuthology.orchestra.run.smithi078.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.62 s, remaining 2.36 s) 2024-09-16T01:11:11.763 INFO:teuthology.orchestra.run.smithi078.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.71 s, remaining 2.20 s) 2024-09-16T01:11:11.913 INFO:teuthology.orchestra.run.smithi078.stderr:1500000 of 3200000 tuples (46%) done (elapsed 1.86 s, remaining 2.11 s) 2024-09-16T01:11:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.00 s, remaining 2.00 s) 2024-09-16T01:11:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.09 s, remaining 1.84 s) 2024-09-16T01:11:12.290 INFO:teuthology.orchestra.run.smithi078.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.24 s, remaining 1.74 s) 2024-09-16T01:11:12.460 INFO:teuthology.orchestra.run.smithi078.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.41 s, remaining 1.65 s) 2024-09-16T01:11:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.56 s, remaining 1.54 s) 2024-09-16T01:11:12.697 INFO:teuthology.orchestra.run.smithi078.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.65 s, remaining 1.39 s) 2024-09-16T01:11:12.848 INFO:teuthology.orchestra.run.smithi078.stderr:2200000 of 3200000 tuples (68%) done (elapsed 2.80 s, remaining 1.27 s) 2024-09-16T01:11:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2300000 of 3200000 tuples (71%) done (elapsed 2.88 s, remaining 1.13 s) 2024-09-16T01:11:13.071 INFO:teuthology.orchestra.run.smithi078.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.02 s, remaining 1.01 s) 2024-09-16T01:11:13.232 INFO:teuthology.orchestra.run.smithi078.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.18 s, remaining 0.89 s) 2024-09-16T01:11:13.325 INFO:teuthology.orchestra.run.smithi078.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.28 s, remaining 0.76 s) 2024-09-16T01:11:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:13 smithi148 ceph-mon[35030]: pgmap v15: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 171 KiB/s wr, 31 op/s 2024-09-16T01:11:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:13 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.40 s, remaining 0.63 s) 2024-09-16T01:11:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:13 smithi078 ceph-mon[30937]: pgmap v15: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 171 KiB/s wr, 31 op/s 2024-09-16T01:11:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:13 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:13 smithi193 ceph-mon[35701]: pgmap v15: 129 pgs: 129 active+clean; 2.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 171 KiB/s wr, 31 op/s 2024-09-16T01:11:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:13 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.50 s, remaining 0.50 s) 2024-09-16T01:11:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.65 s, remaining 0.38 s) 2024-09-16T01:11:13.828 INFO:teuthology.orchestra.run.smithi078.stderr:3000000 of 3200000 tuples (93%) done (elapsed 3.78 s, remaining 0.25 s) 2024-09-16T01:11:13.920 INFO:teuthology.orchestra.run.smithi078.stderr:3100000 of 3200000 tuples (96%) done (elapsed 3.87 s, remaining 0.12 s) 2024-09-16T01:11:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.02 s, remaining 0.00 s) 2024-09-16T01:11:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:vacuuming... 2024-09-16T01:11:14.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:14 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:14.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:14 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:14.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:14 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:11:14.700 INFO:teuthology.orchestra.run.smithi078.stderr:creating primary keys... 2024-09-16T01:11:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:15 smithi148 ceph-mon[35030]: pgmap v16: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 46 MiB/s wr, 145 op/s 2024-09-16T01:11:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:15 smithi078 ceph-mon[30937]: pgmap v16: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 46 MiB/s wr, 145 op/s 2024-09-16T01:11:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:15 smithi193 ceph-mon[35701]: pgmap v16: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 46 MiB/s wr, 145 op/s 2024-09-16T01:11:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:done in 5.88 s (drop tables 0.00 s, create tables 0.04 s, client-side generate 4.09 s, vacuum 0.60 s, primary keys 1.16 s). 2024-09-16T01:11:15.862 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-09-16T01:11:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:starting vacuum...end. 2024-09-16T01:11:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/1897641506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f381009f0 msgr2=0x7f6f38100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/1897641506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f381009f0 0x7f6f38100e50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009900 tx=0x7f6f2002a400 comp rx=0 tx=0).stop 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/1897641506 shutdown_connections 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/1897641506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 0x7f6f3810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/1897641506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f381009f0 0x7f6f38100e50 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009900 tx=0x7f6f2002a400 comp rx=0 tx=0).stop 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/1897641506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f38106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.210+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/1897641506 >> 172.21.15.78:0/1897641506 conn(0x7f6f380fc6b0 msgr2=0x7f6f380fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.212+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/1897641506 shutdown_connections 2024-09-16T01:11:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.212+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/1897641506 wait complete. 2024-09-16T01:11:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.213+0000 7f6f3e6dd640 1 Processor -- start 2024-09-16T01:11:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.213+0000 7f6f3e6dd640 1 -- start start 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.213+0000 7f6f3e6dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 0x7f6f3810bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.213+0000 7f6f3e6dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f3e6dd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f3810c530 0x7f6f381a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f3e6dd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38110e00 con 0x7f6f38101390 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f3e6dd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38110c80 con 0x7f6f3810c530 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f3e6dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38110f80 con 0x7f6f38106a00 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34738/0 (socket says 172.21.15.78:34738) 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/611994440 learned_addr learned my addr 172.21.15.78:0/611994440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f3cc7a640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f3810c530 0x7f6f381a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f37fff640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 0x7f6f3810bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/611994440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f3810c530 msgr2=0x7f6f381a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f3810c530 0x7f6f381a6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 msgr2=0x7f6f3810bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 0x7f6f3810bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.214+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/611994440 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f381a6e40 con 0x7f6f38106a00 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.215+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f20004770 tx=0x7f6f20003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.215+0000 7f6f35ffb640 1 -- 172.21.15.78:0/611994440 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2003f020 con 0x7f6f38106a00 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.215+0000 7f6f35ffb640 1 -- 172.21.15.78:0/611994440 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f20031c20 con 0x7f6f38106a00 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.215+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f38113770 con 0x7f6f38106a00 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.215+0000 7f6f35ffb640 1 -- 172.21.15.78:0/611994440 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20035070 con 0x7f6f38106a00 2024-09-16T01:11:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.216+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f38113c10 con 0x7f6f38106a00 2024-09-16T01:11:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.219+0000 7f6f41474740 1 Processor -- start 2024-09-16T01:11:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 -- start start 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeb0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eeb0ca0 0x55935eeb3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55935ee0a2f0 con 0x55935eea6400 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55935edb2270 con 0x55935eeb0ca0 2024-09-16T01:11:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.220+0000 7f6f41474740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55935ee062b0 con 0x55935eea5ae0 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeb0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34746/0 (socket says 172.21.15.78:34746) 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeb0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55494/0 (socket says 172.21.15.78:55494) 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 -- 172.21.15.78:0/3740341227 learned_addr learned my addr 172.21.15.78:0/3740341227 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f35ffb640 1 -- 172.21.15.78:0/611994440 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f6f20042020 con 0x7f6f38106a00 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 -- 172.21.15.78:0/3740341227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eeb0ca0 msgr2=0x55935eeb3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eeb0ca0 0x55935eeb3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 -- 172.21.15.78:0/3740341227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 msgr2=0x55935eeb0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeb0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 -- 172.21.15.78:0/3740341227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55935edb1340 con 0x55935eea5ae0 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.221+0000 7f6f3cc7a640 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000b700 tx=0x7f6f3000bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b47ab01cf3a34af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.222+0000 7f6f2e7fc640 1 -- 172.21.15.78:0/3740341227 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30003dd0 con 0x55935eea5ae0 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.222+0000 7f6f2e7fc640 1 -- 172.21.15.78:0/3740341227 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f30015070 con 0x55935eea5ae0 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.222+0000 7f6f35ffb640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 0x7f6f240a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f35ffb640 1 -- 172.21.15.78:0/611994440 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6f200f0a70 con 0x7f6f38106a00 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f2e7fc640 1 -- 172.21.15.78:0/3740341227 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30010730 con 0x55935eea5ae0 2024-09-16T01:11:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f37fff640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 0x7f6f240a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f41474740 1 -- 172.21.15.78:0/3740341227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 msgr2=0x55935eea5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f41474740 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000b700 tx=0x7f6f3000bbc0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.223+0000 7f6f41474740 1 -- 172.21.15.78:0/3740341227 shutdown_connections 2024-09-16T01:11:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eeb0ca0 0x55935eeb3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeb0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 --2- 172.21.15.78:0/3740341227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eea5ec0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 -- 172.21.15.78:0/3740341227 >> 172.21.15.78:0/3740341227 conn(0x55935eea2050 msgr2=0x55935eea2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 -- 172.21.15.78:0/3740341227 shutdown_connections 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 -- 172.21.15.78:0/3740341227 wait complete. 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f37fff640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 0x7f6f240a5b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x55935eea2eb0 tx=0x7f6f28009680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 -- start start 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.224+0000 7f6f41474740 1 Processor -- start 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eeb0ca0 0x55935eeae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55935eeb3910 con 0x55935eea6400 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55935ed93290 con 0x55935eea5ae0 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55935eeb3a90 con 0x55935eeb0ca0 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f41474740 10 client.0 _ll_register_callbacks cb 0x55935edc4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:60476/0 (socket says 172.21.15.78:60476) 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 -- 172.21.15.78:0/3511226763 learned_addr learned my addr 172.21.15.78:0/3511226763 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f3cc7a640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eeb0ca0 0x55935eeae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 -- 172.21.15.78:0/3511226763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eeb0ca0 msgr2=0x55935eeae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eeb0ca0 0x55935eeae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 -- 172.21.15.78:0/3511226763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 msgr2=0x55935eeae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f37fff640 1 -- 172.21.15.78:0/3511226763 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55935eb1d210 con 0x55935eea5ae0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.225+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.226+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f28003f20 tx=0x7f6f28004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405dd6811fe54ce9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.226+0000 7f6f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.226+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2801f9b0 con 0x55935eea5ae0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.226+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f30015070 con 0x55935eea5ae0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.226+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6f2801fb50 con 0x55935eea5ae0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.227+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f2801fd50 con 0x55935eea5ae0 2024-09-16T01:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.229+0000 7f6f41474740 10 client.24980 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.229+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55935ebe9620 con 0x55935eea5ae0 2024-09-16T01:11:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.229+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f280278e0 con 0x55935eea5ae0 2024-09-16T01:11:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.229+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f6f2800f040 con 0x55935eea5ae0 2024-09-16T01:11:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 10 client.24980 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 10 client.24980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 20 client.24980 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935eec0d00 0x55935eec30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 4 client.24980 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6f30015070 con 0x55935eec0d00 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.230+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935eec0d00 0x55935eec30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.231+0000 7f6f177fe640 10 client.24980 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.231+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935eec0d00 0x55935eec30f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bb1581801587f77 server_cookie=96ae2cf9af5f545f in_seq=0 out_seq=0 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f177fe640 1 -- 172.21.15.78:0/3511226763 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f6f30015070 con 0x55935eec0d00 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f177fe640 10 client.24980 handle_command_reply: tid=0 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 client.24980 shutdown 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 msgr2=0x55935eeadb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f28003f20 tx=0x7f6f28004a40 comp rx=0 tx=0).stop 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 shutdown_connections 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935eec0d00 0x55935eec30f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f177fe640 0 client.24980 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eeb0ca0 0x55935eeae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eea6400 0x55935eeae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 --2- 172.21.15.78:0/3511226763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea5ae0 0x55935eeadb40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.232+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 >> 172.21.15.78:0/3511226763 conn(0x55935eea2050 msgr2=0x55935eea7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.235+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 shutdown_connections 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.235+0000 7f6f41474740 1 -- 172.21.15.78:0/3511226763 wait complete. 2024-09-16T01:11:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.235+0000 7f6f41474740 20 client.24980 trim_cache size 0 max 16384 2024-09-16T01:11:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.262+0000 7f6f41474740 1 Processor -- start 2024-09-16T01:11:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 -- start start 2024-09-16T01:11:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eec8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eec8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b8480 con 0x55935eeb0ca0 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b5d20 con 0x55935eea6400 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f41474740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b3b70 con 0x55935eea5ae0 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f3cc7a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eec8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eec8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f3cc7a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eec8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55510/0 (socket says 172.21.15.78:55510) 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f3cc7a640 1 -- 172.21.15.78:0/2673144767 learned_addr learned my addr 172.21.15.78:0/2673144767 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f37fff640 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/2673144767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 msgr2=0x55935eeae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/2673144767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 msgr2=0x55935eec8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eec8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.263+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/2673144767 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55935eea6230 con 0x55935eea6400 2024-09-16T01:11:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.264+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eec8a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f200edbf0 tx=0x7f6f20003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f140ecedf7672f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.265+0000 7f6f2dffb640 1 -- 172.21.15.78:0/2673144767 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f200bce80 con 0x55935eea6400 2024-09-16T01:11:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.266+0000 7f6f2dffb640 1 -- 172.21.15.78:0/2673144767 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f200bc780 con 0x55935eea6400 2024-09-16T01:11:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.266+0000 7f6f2dffb640 1 -- 172.21.15.78:0/2673144767 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f200b8df0 con 0x55935eea6400 2024-09-16T01:11:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 -- 172.21.15.78:0/2673144767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 msgr2=0x55935eec8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eec8a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6f200edbf0 tx=0x7f6f20003f90 comp rx=0 tx=0).stop 2024-09-16T01:11:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 -- 172.21.15.78:0/2673144767 shutdown_connections 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eec8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eec8a00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 --2- 172.21.15.78:0/2673144767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.267+0000 7f6f41474740 1 -- 172.21.15.78:0/2673144767 >> 172.21.15.78:0/2673144767 conn(0x55935eea2050 msgr2=0x55935eeafbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.269+0000 7f6f41474740 1 -- 172.21.15.78:0/2673144767 shutdown_connections 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.269+0000 7f6f41474740 1 -- 172.21.15.78:0/2673144767 wait complete. 2024-09-16T01:11:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.269+0000 7f6f41474740 1 -- start start 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.269+0000 7f6f41474740 1 Processor -- start 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeb9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eeb2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b41e0 con 0x55935eeb0ca0 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b5650 con 0x55935eea6400 2024-09-16T01:11:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381b48b0 con 0x55935eea5ae0 2024-09-16T01:11:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f41474740 10 client.0 _ll_register_callbacks cb 0x7f6f28009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f2f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:60504/0 (socket says 172.21.15.78:60504) 2024-09-16T01:11:17.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.270+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/3503283458 learned_addr learned my addr 172.21.15.78:0/3503283458 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeb9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/3503283458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 msgr2=0x55935eeb9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeb9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/3503283458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 msgr2=0x55935eeb2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eeb2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 -- 172.21.15.78:0/3503283458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55935eeb2890 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f2f7fe640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f200096c0 tx=0x7f6f20005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98977f87a24a96e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.271+0000 7f6f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f200bce80 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f6f200b8df0 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f200bc780 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f200baa80 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f41474740 10 client.24986 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.272+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55935ef2b9d0 con 0x55935eea6400 2024-09-16T01:11:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.273+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f200bc780 con 0x55935eea6400 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f6f200b8040 con 0x55935eea6400 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 10 client.24986 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 10 client.24986 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 20 client.24986 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935ef2c010 0x55935ef2e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 4 client.24986 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:11:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.274+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55935ee17a20 con 0x55935ef2c010 2024-09-16T01:11:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.275+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935ef2c010 0x55935ef2e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.276+0000 7f6f37fff640 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935ef2c010 0x55935ef2e430 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b251dae56eb3d70 server_cookie=6aefcf1b204e64bd in_seq=0 out_seq=0 2024-09-16T01:11:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.276+0000 7f6f177fe640 10 client.24986 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.285+0000 7f6f177fe640 1 -- 172.21.15.78:0/3503283458 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55935ee17a20 con 0x55935ef2c010 2024-09-16T01:11:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.285+0000 7f6f177fe640 10 client.24986 handle_command_reply: tid=0 2024-09-16T01:11:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.286+0000 7f6f41474740 1 client.24986 shutdown 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 msgr2=0x55935eeb1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f200096c0 tx=0x7f6f20005cb0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 shutdown_connections 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55935ef2c010 0x55935ef2e430 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935eeb0ca0 0x55935eeb2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55935eea6400 0x55935eeb1b40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 --2- 172.21.15.78:0/3503283458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55935eea5ae0 0x55935eeb9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f177fe640 0 client.24986 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 >> 172.21.15.78:0/3503283458 conn(0x55935eea2050 msgr2=0x55935eea09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 shutdown_connections 2024-09-16T01:11:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 1 -- 172.21.15.78:0/3503283458 wait complete. 2024-09-16T01:11:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.288+0000 7f6f41474740 20 client.24986 trim_cache size 0 max 16384 2024-09-16T01:11:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:11:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:11:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:11:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:11:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.291+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 msgr2=0x7f6f240a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.291+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 0x7f6f240a5b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x55935eea2eb0 tx=0x7f6f28009680 comp rx=0 tx=0).stop 2024-09-16T01:11:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.291+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 msgr2=0x7f6f3810bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.291+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6f20004770 tx=0x7f6f20003b90 comp rx=0 tx=0).stop 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 shutdown_connections 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6f240a3690 0x7f6f240a5b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f3810c530 0x7f6f381a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f38106a00 0x7f6f3810bff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 --2- 172.21.15.78:0/611994440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f38101390 0x7f6f3810bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 >> 172.21.15.78:0/611994440 conn(0x7f6f380fc6b0 msgr2=0x7f6f380fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.292+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 shutdown_connections 2024-09-16T01:11:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:17.293+0000 7f6f3e6dd640 1 -- 172.21.15.78:0/611994440 wait complete. 2024-09-16T01:11:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:17 smithi148 ceph-mon[35030]: pgmap v17: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:17.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:17 smithi078 ceph-mon[30937]: pgmap v17: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:17 smithi193 ceph-mon[35701]: pgmap v17: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:19.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:19 smithi078 ceph-mon[30937]: pgmap v18: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:19 smithi193 ceph-mon[35701]: pgmap v18: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:19.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:19 smithi148 ceph-mon[35030]: pgmap v18: 129 pgs: 129 active+clean; 297 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 46 MiB/s wr, 127 op/s 2024-09-16T01:11:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 5.0 s, 3335.8 tps, lat 29.447 ms stddev 31.121 2024-09-16T01:11:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:21 smithi078 ceph-mon[30937]: pgmap v19: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 76 MiB/s wr, 206 op/s 2024-09-16T01:11:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:21 smithi193 ceph-mon[35701]: pgmap v19: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 76 MiB/s wr, 206 op/s 2024-09-16T01:11:21.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:21 smithi148 ceph-mon[35030]: pgmap v19: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 76 MiB/s wr, 206 op/s 2024-09-16T01:11:23.437 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:23 smithi148 ceph-mon[35030]: pgmap v20: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 76 MiB/s wr, 193 op/s 2024-09-16T01:11:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:23 smithi078 ceph-mon[30937]: pgmap v20: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 76 MiB/s wr, 193 op/s 2024-09-16T01:11:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:23 smithi193 ceph-mon[35701]: pgmap v20: 129 pgs: 129 active+clean; 514 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 76 MiB/s wr, 193 op/s 2024-09-16T01:11:25.462 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-09-16T01:11:25.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:11:25.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:25 smithi078 ceph-mon[30937]: pgmap v21: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 79 MiB/s wr, 466 op/s 2024-09-16T01:11:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:25 smithi193 ceph-mon[35701]: pgmap v21: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 79 MiB/s wr, 466 op/s 2024-09-16T01:11:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:25 smithi148 ceph-mon[35030]: pgmap v21: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 79 MiB/s wr, 466 op/s 2024-09-16T01:11:25.855 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 10.0 s, 3283.0 tps, lat 30.536 ms stddev 33.522 2024-09-16T01:11:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 -- 172.21.15.78:0/268437676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4105e00 msgr2=0x7f4bb410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/268437676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4105e00 0x7f4bb410d9a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009cd0 tx=0x7f4bac02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 -- 172.21.15.78:0/268437676 shutdown_connections 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/268437676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4105e00 0x7f4bb410d9a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/268437676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb41021b0 0x7f4bb41056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/268437676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb4101800 0x7f4bb4101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 -- 172.21.15.78:0/268437676 >> 172.21.15.78:0/268437676 conn(0x7f4bb40fba90 msgr2=0x7f4bb40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 -- 172.21.15.78:0/268437676 shutdown_connections 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.501+0000 7f4bbbe66640 1 -- 172.21.15.78:0/268437676 wait complete. 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 Processor -- start 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 -- start start 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 0x7f4bb41992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb41021b0 0x7f4bb4199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 0x7f4bb41a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4110e50 con 0x7f4bb4101800 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4110cd0 con 0x7f4bb4105e00 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bbbe66640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4110fd0 con 0x7f4bb41021b0 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bb9c02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 0x7f4bb41992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bb9c02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 0x7f4bb41992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51632/0 (socket says 172.21.15.78:51632) 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bb9c02640 1 -- 172.21.15.78:0/755071132 learned_addr learned my addr 172.21.15.78:0/755071132 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bba403640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 0x7f4bb41a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.502+0000 7f4bb9401640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb41021b0 0x7f4bb4199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 -- 172.21.15.78:0/755071132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb41021b0 msgr2=0x7f4bb4199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb41021b0 0x7f4bb4199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 -- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 msgr2=0x7f4bb41992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 0x7f4bb41992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 -- 172.21.15.78:0/755071132 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb41a48a0 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.503+0000 7f4bba403640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 0x7f4bb41a4360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4bac02ae00 tx=0x7f4bac003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.504+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac04c070 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.504+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bac03c720 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.504+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac004420 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.504+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb41970d0 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.504+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb41975c0 con 0x7f4bb4105e00 2024-09-16T01:11:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.505+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb4100070 con 0x7f4bb4105e00 2024-09-16T01:11:26.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.507+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f4bac0045c0 con 0x7f4bb4105e00 2024-09-16T01:11:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.508+0000 7f4baaffd640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 0x7f4ba00a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.508+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4bac02e020 con 0x7f4bb4105e00 2024-09-16T01:11:26.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.509+0000 7f4bb9c02640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 0x7f4ba00a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:26.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.510+0000 7f4bb9c02640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 0x7f4ba00a5b70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000bca0 tx=0x7f4bb000b410 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:26.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.513+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4bac0c0d60 con 0x7f4bb4105e00 2024-09-16T01:11:26.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.872+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bb4108fd0 con 0x7f4bb4105e00 2024-09-16T01:11:26.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.872+0000 7f4baaffd640 1 -- 172.21.15.78:0/755071132 <== mon.2 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11922 (secure 0 0 0) 0x7f4bac0c0660 con 0x7f4bb4105e00 2024-09-16T01:11:26.874 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:26.874 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":28,"btime":"2024-09-16T01:11:13:114118+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:11:12.780839+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:11:26.874 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 28 2024-09-16T01:11:26.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.875+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 msgr2=0x7f4ba00a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:26.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.875+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 0x7f4ba00a5b70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000bca0 tx=0x7f4bb000b410 comp rx=0 tx=0).stop 2024-09-16T01:11:26.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.875+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 msgr2=0x7f4bb41a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 0x7f4bb41a4360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4bac02ae00 tx=0x7f4bac003970 comp rx=0 tx=0).stop 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 shutdown_connections 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4ba00a36b0 0x7f4ba00a5b70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bb4105e00 0x7f4bb41a4360 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bb41021b0 0x7f4bb4199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 --2- 172.21.15.78:0/755071132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4bb4101800 0x7f4bb41992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 >> 172.21.15.78:0/755071132 conn(0x7f4bb40fba90 msgr2=0x7f4bb40fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 shutdown_connections 2024-09-16T01:11:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:26.876+0000 7f4bbbe66640 1 -- 172.21.15.78:0/755071132 wait complete. 2024-09-16T01:11:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:27 smithi078 ceph-mon[30937]: pgmap v22: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:27 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/755071132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:27 smithi193 ceph-mon[35701]: pgmap v22: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:27 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/755071132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:27 smithi148 ceph-mon[35030]: pgmap v22: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:27 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/755071132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:27.682 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0' 2024-09-16T01:11:27.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0 2024-09-16T01:11:28.106 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:28.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.684+0000 7f4622a84640 1 -- 172.21.15.78:0/1150610848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 msgr2=0x7f461c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.684+0000 7f4622a84640 1 --2- 172.21.15.78:0/1150610848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1021c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4610009900 tx=0x7f461002a410 comp rx=0 tx=0).stop 2024-09-16T01:11:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 -- 172.21.15.78:0/1150610848 shutdown_connections 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 --2- 172.21.15.78:0/1150610848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c106f10 0x7f461c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 --2- 172.21.15.78:0/1150610848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c102790 0x7f461c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 --2- 172.21.15.78:0/1150610848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1021c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 -- 172.21.15.78:0/1150610848 >> 172.21.15.78:0/1150610848 conn(0x7f461c0fd8e0 msgr2=0x7f461c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.685+0000 7f4622a84640 1 -- 172.21.15.78:0/1150610848 shutdown_connections 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.686+0000 7f4622a84640 1 -- 172.21.15.78:0/1150610848 wait complete. 2024-09-16T01:11:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.686+0000 7f4622a84640 1 Processor -- start 2024-09-16T01:11:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.686+0000 7f4622a84640 1 -- start start 2024-09-16T01:11:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 0x7f461c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f461bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f461bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43250/0 (socket says 172.21.15.78:43250) 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4620820640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4620820640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:45550/0 (socket says 172.21.15.78:45550) 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f461c111f70 con 0x7f461c106f10 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f461c111df0 con 0x7f461c102790 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f4622a84640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1120f0 con 0x7f461c101de0 2024-09-16T01:11:28.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.687+0000 7f461bfff640 1 -- 172.21.15.78:0/1505968475 learned_addr learned my addr 172.21.15.78:0/1505968475 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 -- 172.21.15.78:0/1505968475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 msgr2=0x7f461c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 -- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 msgr2=0x7f461c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f4621021640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 0x7f461c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 0x7f461c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 -- 172.21.15.78:0/1505968475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c19d460 con 0x7f461c102790 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f4621021640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 0x7f461c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f461bfff640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f460800e9f0 tx=0x7f460800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f4619ffb640 1 -- 172.21.15.78:0/1505968475 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460800cd80 con 0x7f461c102790 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c19d750 con 0x7f461c102790 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.688+0000 7f4619ffb640 1 -- 172.21.15.78:0/1505968475 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608010910 con 0x7f461c102790 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.689+0000 7f4619ffb640 1 -- 172.21.15.78:0/1505968475 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46080188f0 con 0x7f461c102790 2024-09-16T01:11:28.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.689+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c074420 con 0x7f461c102790 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 Processor -- start 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 -- start start 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402d940 0x55f4d402fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f4d3f882a0 con 0x55f4d4023080 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f4d3f2fb10 con 0x55f4d402d940 2024-09-16T01:11:28.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4624e5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f4d3f84260 con 0x55f4d4022760 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4621021640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4621021640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:45562/0 (socket says 172.21.15.78:45562) 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4621021640 1 -- 172.21.15.78:0/769900569 learned_addr learned my addr 172.21.15.78:0/769900569 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.693+0000 7f4620820640 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 -- 172.21.15.78:0/769900569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402d940 msgr2=0x55f4d402fd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402d940 0x55f4d402fd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 -- 172.21.15.78:0/769900569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 msgr2=0x55f4d402d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 -- 172.21.15.78:0/769900569 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d3cf9b90 con 0x55f4d4022760 2024-09-16T01:11:28.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4620820640 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4621021640 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4614009ee0 tx=0x7f461400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3935fb3f2930b5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4606ffd640 1 -- 172.21.15.78:0/769900569 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614011640 con 0x55f4d4022760 2024-09-16T01:11:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4606ffd640 1 -- 172.21.15.78:0/769900569 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4614011cc0 con 0x55f4d4022760 2024-09-16T01:11:28.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.694+0000 7f4606ffd640 1 -- 172.21.15.78:0/769900569 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46140103d0 con 0x55f4d4022760 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.695+0000 7f4624e5d740 1 -- 172.21.15.78:0/769900569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 msgr2=0x55f4d4022b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.695+0000 7f4624e5d740 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4614009ee0 tx=0x7f461400f730 comp rx=0 tx=0).stop 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4624e5d740 1 -- 172.21.15.78:0/769900569 shutdown_connections 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4624e5d740 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402d940 0x55f4d402fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4624e5d740 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4624e5d740 1 --2- 172.21.15.78:0/769900569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4022b40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4624e5d740 1 -- 172.21.15.78:0/769900569 >> 172.21.15.78:0/769900569 conn(0x55f4d401ecf0 msgr2=0x55f4d401f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:28.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.697+0000 7f4619ffb640 1 -- 172.21.15.78:0/1505968475 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f4608010ac0 con 0x7f461c102790 2024-09-16T01:11:28.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.698+0000 7f4619ffb640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 0x7f460c0a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.698+0000 7f4619ffb640 1 -- 172.21.15.78:0/1505968475 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f46080d0760 con 0x7f461c102790 2024-09-16T01:11:28.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.698+0000 7f4624e5d740 1 -- 172.21.15.78:0/769900569 shutdown_connections 2024-09-16T01:11:28.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.698+0000 7f4624e5d740 1 -- 172.21.15.78:0/769900569 wait complete. 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.699+0000 7f4624e5d740 1 -- start start 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.699+0000 7f4624e5d740 1 Processor -- start 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d402a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f4d4030510 con 0x55f4d4023080 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f4d3f106e0 con 0x55f4d402b410 2024-09-16T01:11:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f4d4030690 con 0x55f4d4022760 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4624e5d740 10 client.0 _ll_register_callbacks cb 0x55f4d3f34a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4621021640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4621021640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43264/0 (socket says 172.21.15.78:43264) 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4621021640 1 -- 172.21.15.78:0/4217126340 learned_addr learned my addr 172.21.15.78:0/4217126340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.700+0000 7f4620820640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d402a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 -- 172.21.15.78:0/4217126340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 msgr2=0x55f4d402a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d402a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 -- 172.21.15.78:0/4217126340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 msgr2=0x55f4d402aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 -- 172.21.15.78:0/4217126340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d40309d0 con 0x55f4d402b410 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.701+0000 7f4621021640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f461401bb70 tx=0x7f461401b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a5e7f21ba93f3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.702+0000 7f4620820640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 0x7f460c0a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.704+0000 7f45fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:11:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.704+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461404b060 con 0x55f4d402b410 2024-09-16T01:11:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.704+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4614011cc0 con 0x55f4d402b410 2024-09-16T01:11:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.704+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4614004c40 con 0x55f4d402b410 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.706+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4614052050 con 0x55f4d402b410 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.706+0000 7f4624e5d740 10 client.25001 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.706+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d4033320 con 0x55f4d402b410 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4614003e50 con 0x55f4d402b410 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f4614004030 con 0x55f4d402b410 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4620820640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 0x7f460c0a5b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4610009230 tx=0x7f461003f040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 10 client.25001 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 10 client.25001 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:11:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 20 client.25001 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d4034b40 0x55f4d4036f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 4 client.25001 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.707+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4614011cc0 con 0x55f4d4034b40 2024-09-16T01:11:28.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.710+0000 7f4620820640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d4034b40 0x55f4d4036f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.710+0000 7f4620820640 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d4034b40 0x55f4d4036f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49e951bd4246aa0b server_cookie=3352309864a0cd86 in_seq=0 out_seq=0 2024-09-16T01:11:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.710+0000 7f45fbfff640 10 client.25001 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.712+0000 7f45fbfff640 1 -- 172.21.15.78:0/4217126340 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f4614011cc0 con 0x55f4d4034b40 2024-09-16T01:11:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.712+0000 7f45fbfff640 10 client.25001 handle_command_reply: tid=0 2024-09-16T01:11:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.712+0000 7f4624e5d740 1 client.25001 shutdown 2024-09-16T01:11:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.712+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 msgr2=0x55f4d4042190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.712+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f461401bb70 tx=0x7f461401b640 comp rx=0 tx=0).stop 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 shutdown_connections 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d4034b40 0x55f4d4036f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4042190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 --2- 172.21.15.78:0/4217126340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d402a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 >> 172.21.15.78:0/4217126340 conn(0x55f4d401ecf0 msgr2=0x55f4d4024a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:28.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.713+0000 7f45fbfff640 0 client.25001 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:28.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.714+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 shutdown_connections 2024-09-16T01:11:28.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.714+0000 7f4624e5d740 1 -- 172.21.15.78:0/4217126340 wait complete. 2024-09-16T01:11:28.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.715+0000 7f4624e5d740 20 client.25001 trim_cache size 0 max 16384 2024-09-16T01:11:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 Processor -- start 2024-09-16T01:11:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 -- start start 2024-09-16T01:11:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4047f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1af970 con 0x55f4d4023080 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1ad210 con 0x55f4d402b410 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4624e5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1ab060 con 0x55f4d4022760 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4620820640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4047f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43268/0 (socket says 172.21.15.78:43268) 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 -- 172.21.15.78:0/3925640195 learned_addr learned my addr 172.21.15.78:0/3925640195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f461bfff640 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 -- 172.21.15.78:0/3925640195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 msgr2=0x55f4d4047f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4047f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 -- 172.21.15.78:0/3925640195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 msgr2=0x55f4d402c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.745+0000 7f4621021640 1 -- 172.21.15.78:0/3925640195 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d401a260 con 0x55f4d402b410 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.746+0000 7f4621021640 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4614009d20 tx=0x7f461400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3f8c50992d7913 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.746+0000 7f46067fc640 1 -- 172.21.15.78:0/3925640195 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461404b070 con 0x55f4d402b410 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.746+0000 7f46067fc640 1 -- 172.21.15.78:0/3925640195 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4614009820 con 0x55f4d402b410 2024-09-16T01:11:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.746+0000 7f46067fc640 1 -- 172.21.15.78:0/3925640195 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614003970 con 0x55f4d402b410 2024-09-16T01:11:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.747+0000 7f4624e5d740 1 -- 172.21.15.78:0/3925640195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 msgr2=0x55f4d4027bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.747+0000 7f4624e5d740 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4614009d20 tx=0x7f461400f730 comp rx=0 tx=0).stop 2024-09-16T01:11:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.748+0000 7f4624e5d740 1 -- 172.21.15.78:0/3925640195 shutdown_connections 2024-09-16T01:11:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.748+0000 7f4624e5d740 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d4027bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.748+0000 7f4624e5d740 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4023080 0x55f4d402c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.748+0000 7f4624e5d740 1 --2- 172.21.15.78:0/3925640195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4022760 0x55f4d4047f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.748+0000 7f4624e5d740 1 -- 172.21.15.78:0/3925640195 >> 172.21.15.78:0/3925640195 conn(0x55f4d401ecf0 msgr2=0x55f4d402c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.749+0000 7f4624e5d740 1 -- 172.21.15.78:0/3925640195 shutdown_connections 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.749+0000 7f4624e5d740 1 -- 172.21.15.78:0/3925640195 wait complete. 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.749+0000 7f4624e5d740 1 -- start start 2024-09-16T01:11:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.749+0000 7f4624e5d740 1 Processor -- start 2024-09-16T01:11:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4022760 0x55f4d403a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d403b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1ab6d0 con 0x55f4d4022760 2024-09-16T01:11:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1acb40 con 0x55f4d402b410 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1abda0 con 0x55f4d4023080 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f461bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4620820640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4022760 0x55f4d403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f461bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:45584/0 (socket says 172.21.15.78:45584) 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f461bfff640 1 -- 172.21.15.78:0/2944441524 learned_addr learned my addr 172.21.15.78:0/2944441524 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4624e5d740 10 client.0 _ll_register_callbacks cb 0x7f461401b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.750+0000 7f4621021640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d403b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 -- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 msgr2=0x55f4d403b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d403b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 -- 172.21.15.78:0/2944441524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4022760 msgr2=0x55f4d403a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4022760 0x55f4d403a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 -- 172.21.15.78:0/2944441524 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d40a78d0 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f461bfff640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4608098e40 tx=0x7f460809bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a1f3bc868a03416 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f45fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46080b5060 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f46080b5400 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4614009820 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f46080a0820 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f4624e5d740 10 client.25087 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.751+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d40a8bb0 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46080a50e0 con 0x55f4d4023080 2024-09-16T01:11:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f46080981c0 con 0x55f4d4023080 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 10 client.25087 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 10 client.25087 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 20 client.25087 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d40a9380 0x55f4d40ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 4 client.25087 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}] 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}) -- 0x7f4614009820 con 0x55f4d40a9380 2024-09-16T01:11:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.752+0000 7f4620820640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d40a9380 0x55f4d40ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:28.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.753+0000 7f4620820640 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d40a9380 0x55f4d40ab7a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2294a5240f79cf2d server_cookie=edd41d584a588b2c in_seq=0 out_seq=0 2024-09-16T01:11:28.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:28.753+0000 7f45fbfff640 10 client.25087 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:29.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:29 smithi078 ceph-mon[30937]: pgmap v23: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:29 smithi193 ceph-mon[35701]: pgmap v23: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:29.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:29 smithi148 ceph-mon[35030]: pgmap v23: 129 pgs: 129 active+clean; 900 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 34 MiB/s wr, 352 op/s 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f45fbfff640 1 -- 172.21.15.78:0/2944441524 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f4614009820 con 0x55f4d40a9380 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f45fbfff640 10 client.25087 handle_command_reply: tid=0 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 client.25087 shutdown 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 msgr2=0x55f4d403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4608098e40 tx=0x7f460809bec0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 shutdown_connections 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f4d40a9380 0x55f4d40ab7a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4d402b410 0x55f4d403b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f45fbfff640 0 client.25087 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4d4023080 0x55f4d403ad70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 --2- 172.21.15.78:0/2944441524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f4d4022760 0x55f4d403a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.164+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 >> 172.21.15.78:0/2944441524 conn(0x55f4d401ecf0 msgr2=0x55f4d4047e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 shutdown_connections 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4624e5d740 1 -- 172.21.15.78:0/2944441524 wait complete. 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4624e5d740 20 client.25087 trim_cache size 0 max 16384 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 28, 2024-09-16T01:11:30.167 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 2, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "538a64bd": { 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 2, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 150.2, 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:11:30.168 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:30.169 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 msgr2=0x7f460c0a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 0x7f460c0a5b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4610009230 tx=0x7f461003f040 comp rx=0 tx=0).stop 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 msgr2=0x7f461c1989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f460800e9f0 tx=0x7f460800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 shutdown_connections 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f460c0a3660 0x7f460c0a5b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f461c106f10 0x7f461c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f461c102790 0x7f461c1989f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 --2- 172.21.15.78:0/1505968475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101de0 0x7f461c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.165+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 >> 172.21.15.78:0/1505968475 conn(0x7f461c0fd8e0 msgr2=0x7f461c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.166+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 shutdown_connections 2024-09-16T01:11:30.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:30.166+0000 7f4622a84640 1 -- 172.21.15.78:0/1505968475 wait complete. 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.385+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/2485655138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1069d0 msgr2=0x7f8cac106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.385+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/2485655138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1069d0 0x7f8cac106db0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009930 tx=0x7f8c9c02a420 comp rx=0 tx=0).stop 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/2485655138 shutdown_connections 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/2485655138 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac10d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/2485655138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1009c0 0x7f8cac100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/2485655138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1069d0 0x7f8cac106db0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/2485655138 >> 172.21.15.78:0/2485655138 conn(0x7f8cac0fc700 msgr2=0x7f8cac0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/2485655138 shutdown_connections 2024-09-16T01:11:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/2485655138 wait complete. 2024-09-16T01:11:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 Processor -- start 2024-09-16T01:11:30.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.386+0000 7f8cb0a8d640 1 -- start start 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1069d0 0x7f8cac19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac110e10 con 0x7f8cac1009c0 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac110c90 con 0x7f8cac101360 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac110f90 con 0x7f8cac1069d0 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51696/0 (socket says 172.21.15.78:51696) 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 -- 172.21.15.78:0/1842920126 learned_addr learned my addr 172.21.15.78:0/1842920126 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1069d0 0x7f8cac19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 -- 172.21.15.78:0/1842920126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1069d0 msgr2=0x7f8cac19cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1069d0 0x7f8cac19cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 -- 172.21.15.78:0/1842920126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 msgr2=0x7f8cac198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 -- 172.21.15.78:0/1842920126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cac19d340 con 0x7f8cac1009c0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009230 tx=0x7f8c9c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8c9b7fe640 1 -- 172.21.15.78:0/1842920126 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c03f030 con 0x7f8cac1009c0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8c9b7fe640 1 -- 172.21.15.78:0/1842920126 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9c003ad0 con 0x7f8cac1009c0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cac19d5d0 con 0x7f8cac1009c0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8c9b7fe640 1 -- 172.21.15.78:0/1842920126 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c035070 con 0x7f8cac1009c0 2024-09-16T01:11:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.387+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cac1a4ec0 con 0x7f8cac1009c0 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.390+0000 7f8c9b7fe640 1 -- 172.21.15.78:0/1842920126 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f8c9c042030 con 0x7f8cac1009c0 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.390+0000 7f8cb382b740 1 Processor -- start 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 -- start start 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd3ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd48cb0 0x559e1dd4b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559e1dca22f0 con 0x559e1dd3e410 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e1dc4a270 con 0x559e1dd3daf0 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8cb382b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559e1dc9e2b0 con 0x559e1dd48cb0 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51712/0 (socket says 172.21.15.78:51712) 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 -- 172.21.15.78:0/1311435463 learned_addr learned my addr 172.21.15.78:0/1311435463 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd48cb0 0x559e1dd4b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd3ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8c9b7fe640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 0x7f8c940a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 -- 172.21.15.78:0/1311435463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd48cb0 msgr2=0x559e1dd4b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd48cb0 0x559e1dd4b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 -- 172.21.15.78:0/1311435463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 msgr2=0x559e1dd3ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd3ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caa59c640 1 -- 172.21.15.78:0/1311435463 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e1dc49340 con 0x559e1dd3e410 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.391+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 0x7f8c940a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0ed9a0 tx=0x7f8c9c0b8960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c243b7d64ea8fd9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8c9b7fe640 1 -- 172.21.15.78:0/1842920126 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8c9c0f0880 con 0x7f8cac1009c0 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8c98ff9640 1 -- 172.21.15.78:0/1311435463 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c049020 con 0x559e1dd3e410 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8c98ff9640 1 -- 172.21.15.78:0/1311435463 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9c0bcc90 con 0x559e1dd3e410 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8c98ff9640 1 -- 172.21.15.78:0/1311435463 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c09c070 con 0x559e1dd3e410 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8cb382b740 1 -- 172.21.15.78:0/1311435463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 msgr2=0x559e1dd48770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 0x7f8c940a5ad0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4005fd0 tx=0x7f8ca4004a40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.392+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0ed9a0 tx=0x7f8c9c0b8960 comp rx=0 tx=0).stop 2024-09-16T01:11:30.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- 172.21.15.78:0/1311435463 shutdown_connections 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd48cb0 0x559e1dd4b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd48770 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1311435463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd3ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- 172.21.15.78:0/1311435463 >> 172.21.15.78:0/1311435463 conn(0x559e1dd3a060 msgr2=0x559e1dd3a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- 172.21.15.78:0/1311435463 shutdown_connections 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- 172.21.15.78:0/1311435463 wait complete. 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- start start 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 Processor -- start 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd45b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 0x559e1dd465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559e1dca22f0 con 0x559e1dd3e410 2024-09-16T01:11:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e1dc2b290 con 0x559e1dd48cb0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559e1dd4b640 con 0x559e1dd3daf0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8cb382b740 10 client.0 _ll_register_callbacks cb 0x559e1dc5c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8ca9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8ca9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51728/0 (socket says 172.21.15.78:51728) 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8ca9d9b640 1 -- 172.21.15.78:0/2248194053 learned_addr learned my addr 172.21.15.78:0/2248194053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.393+0000 7f8caad9d640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 0x559e1dd465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8caa59c640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd45b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 -- 172.21.15.78:0/2248194053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 msgr2=0x559e1dd45b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd45b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 -- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 msgr2=0x559e1dd465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 0x559e1dd465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 -- 172.21.15.78:0/2248194053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e1d9b5210 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8caad9d640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 0x559e1dd465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009f20 tx=0x7f8ca0011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52699113d7f4ca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8c91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0012920 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8ca0012ac0 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c9c0bcc90 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ca0025050 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8cb382b740 10 client.15501 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e1da81620 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.394+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9c0bcc90 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f8ca000f040 con 0x559e1dd3e410 2024-09-16T01:11:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 10 client.15501 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 10 client.15501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 20 client.15501 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1dd58940 0x559e1dd5ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 4 client.15501 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e1dcafa20 con 0x559e1dd58940 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8caa59c640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1dd58940 0x559e1dd5ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8c91ffb640 10 client.15501 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.395+0000 7f8caa59c640 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1dd58940 0x559e1dd5ad60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881d9b13feec66f2 server_cookie=fc682b79445f92aa in_seq=0 out_seq=0 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8c91ffb640 1 -- 172.21.15.78:0/2248194053 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e1dcafa20 con 0x559e1dd58940 2024-09-16T01:11:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8c91ffb640 10 client.15501 handle_command_reply: tid=0 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 client.15501 shutdown 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 msgr2=0x559e1dd46080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009f20 tx=0x7f8ca0011870 comp rx=0 tx=0).stop 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 shutdown_connections 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1dd58940 0x559e1dd5ad60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd48cb0 0x559e1dd465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd3e410 0x559e1dd46080 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8c91ffb640 0 client.15501 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 --2- 172.21.15.78:0/2248194053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd45b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 >> 172.21.15.78:0/2248194053 conn(0x559e1dd3a060 msgr2=0x559e1dd3fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 shutdown_connections 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 1 -- 172.21.15.78:0/2248194053 wait complete. 2024-09-16T01:11:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.396+0000 7f8cb382b740 20 client.15501 trim_cache size 0 max 16384 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 Processor -- start 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 -- start start 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd443f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3e410 0x559e1dd463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1b1850 con 0x559e1dd48cb0 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1af0f0 con 0x559e1dd3e410 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.410+0000 7f8cb382b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1acf40 con 0x559e1dd3daf0 2024-09-16T01:11:30.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51740/0 (socket says 172.21.15.78:51740) 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 -- 172.21.15.78:0/1981780250 learned_addr learned my addr 172.21.15.78:0/1981780250 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3e410 0x559e1dd463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 -- 172.21.15.78:0/1981780250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 msgr2=0x559e1dd443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 -- 172.21.15.78:0/1981780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3e410 msgr2=0x559e1dd463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3e410 0x559e1dd463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 -- 172.21.15.78:0/1981780250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e1dd40940 con 0x559e1dd48cb0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caa59c640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8caad9d640 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40106e0 tx=0x7f8ca4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f693f673fde66898 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8c93fff640 1 -- 172.21.15.78:0/1981780250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4011580 con 0x559e1dd48cb0 2024-09-16T01:11:30.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8c93fff640 1 -- 172.21.15.78:0/1981780250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca4011c00 con 0x559e1dd48cb0 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.411+0000 7f8c93fff640 1 -- 172.21.15.78:0/1981780250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca401bb50 con 0x559e1dd48cb0 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- 172.21.15.78:0/1981780250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 msgr2=0x559e1dd60630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40106e0 tx=0x7f8ca4016000 comp rx=0 tx=0).stop 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- 172.21.15.78:0/1981780250 shutdown_connections 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd60630 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3e410 0x559e1dd463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- 172.21.15.78:0/1981780250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3daf0 0x559e1dd443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- 172.21.15.78:0/1981780250 >> 172.21.15.78:0/1981780250 conn(0x559e1dd3a060 msgr2=0x559e1dd47bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- 172.21.15.78:0/1981780250 shutdown_connections 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- 172.21.15.78:0/1981780250 wait complete. 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- start start 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 Processor -- start 2024-09-16T01:11:30.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd50ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 0x559e1dd42790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1ad5b0 con 0x559e1dd48cb0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1aea20 con 0x559e1dd3daf0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.412+0000 7f8cb382b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1adc80 con 0x559e1dd3e410 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8cb382b740 10 client.0 _ll_register_callbacks cb 0x7f8ca00097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51742/0 (socket says 172.21.15.78:51742) 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 -- 172.21.15.78:0/3838351221 learned_addr learned my addr 172.21.15.78:0/3838351221 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caa59c640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd50ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 0x559e1dd42790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 -- 172.21.15.78:0/3838351221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 msgr2=0x559e1dd42790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 0x559e1dd42790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 -- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 msgr2=0x559e1dd50ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd50ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 -- 172.21.15.78:0/3838351221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e1dd352d0 con 0x559e1dd48cb0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8ca9d9b640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 0x559e1dd42790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8caad9d640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4013fd0 tx=0x7f8ca401b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5298347fc69f2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8c91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4034020 con 0x559e1dd48cb0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8ca402d040 con 0x559e1dd48cb0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ca4011c00 con 0x559e1dd48cb0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ca4011790 con 0x559e1dd48cb0 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8cb382b740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:30.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e1ddc3300 con 0x559e1dd48cb0 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.413+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca4011c00 con 0x559e1dd48cb0 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f8ca40307b0 con 0x559e1dd48cb0 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 10 client.15507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 20 client.15507 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1ddc3aa0 0x559e1ddc5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 4 client.15507 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e1dca9d70 con 0x559e1ddc3aa0 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8caa59c640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1ddc3aa0 0x559e1ddc5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8c91ffb640 10 client.15507 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:30.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.414+0000 7f8caa59c640 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1ddc3aa0 0x559e1ddc5ec0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9540a8983af92a39 server_cookie=a80785086b428781 in_seq=0 out_seq=0 2024-09-16T01:11:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8c91ffb640 1 -- 172.21.15.78:0/3838351221 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e1dca9d70 con 0x559e1ddc3aa0 2024-09-16T01:11:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8c91ffb640 10 client.15507 handle_command_reply: tid=0 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 client.15507 shutdown 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 msgr2=0x559e1dd42cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4013fd0 tx=0x7f8ca401b630 comp rx=0 tx=0).stop 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 shutdown_connections 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e1ddc3aa0 0x559e1ddc5ec0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e1dd48cb0 0x559e1dd42cd0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e1dd3e410 0x559e1dd42790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8c91ffb640 0 client.15507 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 --2- 172.21.15.78:0/3838351221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e1dd3daf0 0x559e1dd50ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 >> 172.21.15.78:0/3838351221 conn(0x559e1dd3a060 msgr2=0x559e1dd389a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 shutdown_connections 2024-09-16T01:11:30.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 1 -- 172.21.15.78:0/3838351221 wait complete. 2024-09-16T01:11:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.421+0000 7f8cb382b740 20 client.15507 trim_cache size 0 max 16384 2024-09-16T01:11:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:11:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:11:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:11:30.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 msgr2=0x7f8c940a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 0x7f8c940a5ad0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4005fd0 tx=0x7f8ca4004a40 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 msgr2=0x7f8cac198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009230 tx=0x7f8c9c004270 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 shutdown_connections 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8c940a3610 0x7f8c940a5ad0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cac1069d0 0x7f8cac19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac101360 0x7f8cac198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 --2- 172.21.15.78:0/1842920126 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cac1009c0 0x7f8cac198350 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:30.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 >> 172.21.15.78:0/1842920126 conn(0x7f8cac0fc700 msgr2=0x7f8cac06a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 shutdown_connections 2024-09-16T01:11:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:30.422+0000 7f8cb0a8d640 1 -- 172.21.15.78:0/1842920126 wait complete. 2024-09-16T01:11:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 15.0 s, 1873.6 tps, lat 30.372 ms stddev 31.055 2024-09-16T01:11:31.163 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '538a64bd', quiesce took 5.7 seconds. Will release after: 24.5 2024-09-16T01:11:31.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:31 smithi078 ceph-mon[30937]: pgmap v24: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 38 MiB/s wr, 660 op/s 2024-09-16T01:11:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:31 smithi193 ceph-mon[35701]: pgmap v24: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 38 MiB/s wr, 660 op/s 2024-09-16T01:11:31.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:31 smithi148 ceph-mon[35030]: pgmap v24: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 38 MiB/s wr, 660 op/s 2024-09-16T01:11:33.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:33 smithi078 ceph-mon[30937]: pgmap v25: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.4 MiB/s wr, 580 op/s 2024-09-16T01:11:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:33 smithi193 ceph-mon[35701]: pgmap v25: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.4 MiB/s wr, 580 op/s 2024-09-16T01:11:33.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:33 smithi148 ceph-mon[35030]: pgmap v25: 129 pgs: 129 active+clean; 931 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.4 MiB/s wr, 580 op/s 2024-09-16T01:11:35.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:35 smithi078 ceph-mon[30937]: pgmap v26: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 8.1 MiB/s wr, 716 op/s 2024-09-16T01:11:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:35 smithi193 ceph-mon[35701]: pgmap v26: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 8.1 MiB/s wr, 716 op/s 2024-09-16T01:11:35.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:35 smithi148 ceph-mon[35030]: pgmap v26: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 8.1 MiB/s wr, 716 op/s 2024-09-16T01:11:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:11:37.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:37 smithi078 ceph-mon[30937]: pgmap v27: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:37 smithi193 ceph-mon[35701]: pgmap v27: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:37.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:37 smithi148 ceph-mon[35030]: pgmap v27: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:39 smithi078 ceph-mon[30937]: pgmap v28: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:39 smithi193 ceph-mon[35701]: pgmap v28: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:39.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:39 smithi148 ceph-mon[35030]: pgmap v28: 129 pgs: 129 active+clean; 934 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 5.1 MiB/s wr, 444 op/s 2024-09-16T01:11:40.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:11:41.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:41 smithi078 ceph-mon[30937]: pgmap v29: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 20 MiB/s wr, 696 op/s 2024-09-16T01:11:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:41 smithi193 ceph-mon[35701]: pgmap v29: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 20 MiB/s wr, 696 op/s 2024-09-16T01:11:41.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:41 smithi148 ceph-mon[35030]: pgmap v29: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 20 MiB/s wr, 696 op/s 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 -- 172.21.15.78:0/3707169493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 msgr2=0x7fe3c0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 --2- 172.21.15.78:0/3707169493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c0106dc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009930 tx=0x7fe3bc02a420 comp rx=0 tx=0).stop 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 -- 172.21.15.78:0/3707169493 shutdown_connections 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 --2- 172.21.15.78:0/3707169493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c0101370 0x7fe3c010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 --2- 172.21.15.78:0/3707169493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c01009d0 0x7fe3c0100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 --2- 172.21.15.78:0/3707169493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c0106dc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.517+0000 7fe3c756d640 1 -- 172.21.15.78:0/3707169493 >> 172.21.15.78:0/3707169493 conn(0x7fe3c00fc6f0 msgr2=0x7fe3c00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- 172.21.15.78:0/3707169493 shutdown_connections 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- 172.21.15.78:0/3707169493 wait complete. 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 Processor -- start 2024-09-16T01:11:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- start start 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c01009d0 0x7fe3c0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c0101370 0x7fe3c0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0110e20 con 0x7fe3c01069e0 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0110ca0 con 0x7fe3c0101370 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.518+0000 7fe3c756d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0110fa0 con 0x7fe3c01009d0 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59392/0 (socket says 172.21.15.78:59392) 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/359578291 learned_addr learned my addr 172.21.15.78:0/359578291 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5309640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c01009d0 0x7fe3c0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c0101370 0x7fe3c0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/359578291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c01009d0 msgr2=0x7fe3c0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c01009d0 0x7fe3c0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/359578291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c0101370 msgr2=0x7fe3c0198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c0101370 0x7fe3c0198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/359578291 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c019d340 con 0x7fe3c01069e0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800eaa0 tx=0x7fe3b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3b67fc640 1 -- 172.21.15.78:0/359578291 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800cdb0 con 0x7fe3c01069e0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3b67fc640 1 -- 172.21.15.78:0/359578291 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b8005020 con 0x7fe3c01069e0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c019d630 con 0x7fe3c01069e0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3b67fc640 1 -- 172.21.15.78:0/359578291 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80116b0 con 0x7fe3c01069e0 2024-09-16T01:11:43.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.519+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c01a4ec0 con 0x7fe3c01069e0 2024-09-16T01:11:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 Processor -- start 2024-09-16T01:11:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3b67fc640 1 -- 172.21.15.78:0/359578291 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7fe3b8005560 con 0x7fe3c01069e0 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 -- start start 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0c410 0x564a5bf16770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf190a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564a5be702f0 con 0x564a5bf0baf0 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564a5be18270 con 0x564a5bf16cb0 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a5be6c2b0 con 0x564a5bf0c410 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3c4b08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3c4b08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59408/0 (socket says 172.21.15.78:59408) 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3c4b08640 1 -- 172.21.15.78:0/507911058 learned_addr learned my addr 172.21.15.78:0/507911058 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.522+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf190a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 -- 172.21.15.78:0/507911058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0c410 msgr2=0x564a5bf16770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0c410 0x564a5bf16770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 -- 172.21.15.78:0/507911058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 msgr2=0x564a5bf190a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf190a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 -- 172.21.15.78:0/507911058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a5be17340 con 0x564a5bf0baf0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3b67fc640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 0x7fe3ac0a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00125b0 tx=0x7fe3b000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95afaf0f2b5fee11 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3c5309640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 0x7fe3ac0a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/507911058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0005020 con 0x564a5bf0baf0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3b67fc640 1 -- 172.21.15.78:0/359578291 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe3b80d0410 con 0x7fe3c01069e0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/507911058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b00047a0 con 0x564a5bf0baf0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.523+0000 7fe3ab7fe640 1 -- 172.21.15.78:0/507911058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00106c0 con 0x564a5bf0baf0 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/507911058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 msgr2=0x564a5bf0bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3c5309640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 0x7fe3ac0a5b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009900 tx=0x7fe3bc000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00125b0 tx=0x7fe3b000b650 comp rx=0 tx=0).stop 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/507911058 shutdown_connections 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf190a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0c410 0x564a5bf16770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/507911058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf0bed0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/507911058 >> 172.21.15.78:0/507911058 conn(0x564a5bf08060 msgr2=0x564a5bf08440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/507911058 shutdown_connections 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/507911058 wait complete. 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 -- start start 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.524+0000 7fe3ca3e6740 1 Processor -- start 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf140d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf16cb0 0x564a5bf14610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564a5bf19880 con 0x564a5bf0baf0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564a5bdf9290 con 0x564a5bf0c410 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564a5bf19a00 con 0x564a5bf16cb0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 10 client.0 _ll_register_callbacks cb 0x564a5be2a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59420/0 (socket says 172.21.15.78:59420) 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 -- 172.21.15.78:0/2823622459 learned_addr learned my addr 172.21.15.78:0/2823622459 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf16cb0 0x564a5bf14610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 -- 172.21.15.78:0/2823622459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf16cb0 msgr2=0x564a5bf14610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf16cb0 0x564a5bf14610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 -- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 msgr2=0x564a5bf140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 -- 172.21.15.78:0/2823622459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a5bb83210 con 0x564a5bf0baf0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3c5309640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc033000 tx=0x7fe3bc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba6ef34caaa56df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc04d070 con 0x564a5bf0baf0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe3bc003a00 con 0x564a5bf0baf0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.525+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b00047a0 con 0x564a5bf0baf0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3bc059050 con 0x564a5bf0baf0 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 10 client.15519 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a5bc4f620 con 0x564a5bf0baf0 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b00047a0 con 0x564a5bf0baf0 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fe3bc03c440 con 0x564a5bf0baf0 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 10 client.15519 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 20 client.15519 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf26c60 0x564a5bf290b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 4 client.15519 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a5be7da20 con 0x564a5bf26c60 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.526+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf26c60 0x564a5bf290b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.527+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf26c60 0x564a5bf290b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633bb9b53f4cfb5a server_cookie=96d74249c1276da9 in_seq=0 out_seq=0 2024-09-16T01:11:43.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.527+0000 7fe3a8ff9640 10 client.15519 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:43.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:43 smithi078 ceph-mon[30937]: pgmap v30: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 15 MiB/s wr, 388 op/s 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/2823622459 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564a5be7da20 con 0x564a5bf26c60 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3a8ff9640 10 client.15519 handle_command_reply: tid=0 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 client.15519 shutdown 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 msgr2=0x564a5bf13b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc033000 tx=0x7fe3bc02a420 comp rx=0 tx=0).stop 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 shutdown_connections 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf26c60 0x564a5bf290b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf16cb0 0x564a5bf14610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/2823622459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0baf0 0x564a5bf13b90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3a8ff9640 0 client.15519 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 >> 172.21.15.78:0/2823622459 conn(0x564a5bf08060 msgr2=0x564a5bf0dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 shutdown_connections 2024-09-16T01:11:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/2823622459 wait complete. 2024-09-16T01:11:43.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.528+0000 7fe3ca3e6740 20 client.15519 trim_cache size 0 max 16384 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.542+0000 7fe3ca3e6740 1 Processor -- start 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.542+0000 7fe3ca3e6740 1 -- start start 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf15c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf14440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b18d0 con 0x564a5bf16cb0 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01af170 con 0x564a5bf0c410 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01acfc0 con 0x564a5bf0baf0 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59430/0 (socket says 172.21.15.78:59430) 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/1241607311 learned_addr learned my addr 172.21.15.78:0/1241607311 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf14440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/1241607311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 msgr2=0x564a5bf15c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf15c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/1241607311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 msgr2=0x564a5bf14440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf14440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 -- 172.21.15.78:0/1241607311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a5bf12b40 con 0x564a5bf16cb0 2024-09-16T01:11:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80092f0 tx=0x7fe3b80a04d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae9afd53c892c9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3aaffd640 1 -- 172.21.15.78:0/1241607311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b801f040 con 0x564a5bf16cb0 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3aaffd640 1 -- 172.21.15.78:0/1241607311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b80bc0c0 con 0x564a5bf16cb0 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.543+0000 7fe3aaffd640 1 -- 172.21.15.78:0/1241607311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80d00b0 con 0x564a5bf16cb0 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1241607311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 msgr2=0x564a5bf2e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80092f0 tx=0x7fe3b80a04d0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1241607311 shutdown_connections 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf16cb0 0x564a5bf2e9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf0c410 0x564a5bf14440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1241607311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf15c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1241607311 >> 172.21.15.78:0/1241607311 conn(0x564a5bf08060 msgr2=0x564a5bf08440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1241607311 shutdown_connections 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1241607311 wait complete. 2024-09-16T01:11:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 -- start start 2024-09-16T01:11:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.544+0000 7fe3ca3e6740 1 Processor -- start 2024-09-16T01:11:43.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:43 smithi193 ceph-mon[35701]: pgmap v30: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 15 MiB/s wr, 388 op/s 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf1f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf10d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01ad630 con 0x564a5bf0c410 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01aeaa0 con 0x564a5bf16cb0 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01add00 con 0x564a5bf0baf0 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 10 client.0 _ll_register_callbacks cb 0x7fe3bc040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59432/0 (socket says 172.21.15.78:59432) 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 -- 172.21.15.78:0/1528601635 learned_addr learned my addr 172.21.15.78:0/1528601635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c5b0a640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf10d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c5309640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf1f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 -- 172.21.15.78:0/1528601635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 msgr2=0x564a5bf1f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf1f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 -- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 msgr2=0x564a5bf10d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf10d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 -- 172.21.15.78:0/1528601635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a5bf0c240 con 0x564a5bf0c410 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3c4b08640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0012f00 tx=0x7fe3b000ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584ce8ef616f44e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0005450 con 0x564a5bf0c410 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b80bc0c0 con 0x564a5bf0c410 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.545+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe3b00055f0 con 0x564a5bf0c410 2024-09-16T01:11:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3b0012250 con 0x564a5bf0c410 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a5bf91740 con 0x564a5bf0c410 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b80bc0c0 con 0x564a5bf0c410 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fe3b00108b0 con 0x564a5bf0c410 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 20 client.15525 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf91da0 0x564a5bf941c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 4 client.15525 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a5be77d70 con 0x564a5bf91da0 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.546+0000 7fe3c5309640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf91da0 0x564a5bf941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.547+0000 7fe3c5309640 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf91da0 0x564a5bf941c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab82009a19e5b7a1 server_cookie=a463aad9053f238a in_seq=0 out_seq=0 2024-09-16T01:11:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.547+0000 7fe3a8ff9640 10 client.15525 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3a8ff9640 1 -- 172.21.15.78:0/1528601635 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a5be77d70 con 0x564a5bf91da0 2024-09-16T01:11:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3a8ff9640 10 client.15525 handle_command_reply: tid=0 2024-09-16T01:11:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 client.15525 shutdown 2024-09-16T01:11:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 msgr2=0x564a5bf10820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0012f00 tx=0x7fe3b000ba00 comp rx=0 tx=0).stop 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 shutdown_connections 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564a5bf91da0 0x564a5bf941c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564a5bf16cb0 0x564a5bf10d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a5bf0c410 0x564a5bf10820 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 --2- 172.21.15.78:0/1528601635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a5bf0baf0 0x564a5bf1f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3a8ff9640 0 client.15525 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 >> 172.21.15.78:0/1528601635 conn(0x564a5bf08060 msgr2=0x564a5bf071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 shutdown_connections 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 1 -- 172.21.15.78:0/1528601635 wait complete. 2024-09-16T01:11:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.554+0000 7fe3ca3e6740 20 client.15525 trim_cache size 0 max 16384 2024-09-16T01:11:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:11:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:11:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:11:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 msgr2=0x7fe3ac0a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 0x7fe3ac0a5b50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009900 tx=0x7fe3bc000bc0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 msgr2=0x7fe3c019cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800eaa0 tx=0x7fe3b800ef60 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 shutdown_connections 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3ac0a3690 0x7fe3ac0a5b50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3c01069e0 0x7fe3c019cc10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3c0101370 0x7fe3c0198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 --2- 172.21.15.78:0/359578291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3c01009d0 0x7fe3c0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 >> 172.21.15.78:0/359578291 conn(0x7fe3c00fc6f0 msgr2=0x7fe3c01046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 shutdown_connections 2024-09-16T01:11:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:43.555+0000 7fe3c756d640 1 -- 172.21.15.78:0/359578291 wait complete. 2024-09-16T01:11:43.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:43 smithi148 ceph-mon[35030]: pgmap v30: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 15 MiB/s wr, 388 op/s 2024-09-16T01:11:45.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:45 smithi078 ceph-mon[30937]: pgmap v31: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 391 op/s 2024-09-16T01:11:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:45 smithi193 ceph-mon[35701]: pgmap v31: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 391 op/s 2024-09-16T01:11:45.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:45 smithi148 ceph-mon[35030]: pgmap v31: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 391 op/s 2024-09-16T01:11:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:11:46.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:46 smithi078 ceph-mon[30937]: pgmap v32: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:46.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:46 smithi193 ceph-mon[35701]: pgmap v32: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:46.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:46 smithi148 ceph-mon[35030]: pgmap v32: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:48.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:48 smithi193 ceph-mon[35701]: pgmap v33: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:48.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:48 smithi148 ceph-mon[35030]: pgmap v33: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:48.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:48 smithi078 ceph-mon[30937]: pgmap v33: 129 pgs: 129 active+clean; 940 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 255 op/s 2024-09-16T01:11:50.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:50 smithi193 ceph-mon[35701]: pgmap v34: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 396 op/s 2024-09-16T01:11:50.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:50 smithi148 ceph-mon[35030]: pgmap v34: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 396 op/s 2024-09-16T01:11:50.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:50 smithi078 ceph-mon[30937]: pgmap v34: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 396 op/s 2024-09-16T01:11:50.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:11:51.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:51 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:11:51.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:51 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:11:51.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:51 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:11:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: pgmap v35: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 16 MiB/s wr, 143 op/s 2024-09-16T01:11:52.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: pgmap v35: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 16 MiB/s wr, 143 op/s 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.799 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:11:52.800 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:11:52.800 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: pgmap v35: 129 pgs: 129 active+clean; 945 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 16 MiB/s wr, 143 op/s 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:11:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:11:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:54 smithi078 ceph-mon[30937]: pgmap v36: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:11:54.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:54 smithi193 ceph-mon[35701]: pgmap v36: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:11:54.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:54 smithi148 ceph-mon[35030]: pgmap v36: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:11:55.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:11:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 40.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:11:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 -- 172.21.15.78:0/2921374161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 msgr2=0x7f2078100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 --2- 172.21.15.78:0/2921374161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078100ee0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2068009930 tx=0x7f206802a420 comp rx=0 tx=0).stop 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 -- 172.21.15.78:0/2921374161 shutdown_connections 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 --2- 172.21.15.78:0/2921374161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 0x7f207810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 --2- 172.21.15.78:0/2921374161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078100ee0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 --2- 172.21.15.78:0/2921374161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 0x7f2078106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 -- 172.21.15.78:0/2921374161 >> 172.21.15.78:0/2921374161 conn(0x7f20780fc7a0 msgr2=0x7f20780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 -- 172.21.15.78:0/2921374161 shutdown_connections 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.542+0000 7f207fe97640 1 -- 172.21.15.78:0/2921374161 wait complete. 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 Processor -- start 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 -- start start 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 0x7f2078199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 0x7f20781a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2078110ea0 con 0x7f2078100a80 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2078110d20 con 0x7f2078106a80 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207fe97640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2078111020 con 0x7f2078101420 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207dc33640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207dc33640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54272/0 (socket says 172.21.15.78:54272) 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207dc33640 1 -- 172.21.15.78:0/3524059201 learned_addr learned my addr 172.21.15.78:0/3524059201 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.543+0000 7f207e434640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 0x7f20781a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207d432640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 0x7f2078199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 -- 172.21.15.78:0/3524059201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 msgr2=0x7f2078199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 0x7f2078199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 -- 172.21.15.78:0/3524059201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 msgr2=0x7f20781a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 0x7f20781a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 -- 172.21.15.78:0/3524059201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20781a48f0 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207dc33640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f207400c890 tx=0x7f207400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074004220 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20781a4ac0 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20740043c0 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074004a00 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.544+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a5000 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.545+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2078100ee0 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.546+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f20740079b0 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.547+0000 7f206effd640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 0x7f20640a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.547+0000 7f207d432640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 0x7f20640a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.547+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f20740d0080 con 0x7f2078100a80 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.547+0000 7f207d432640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 0x7f20640a5a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20680096c0 tx=0x7f2068000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.551+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2074014030 con 0x7f2078100a80 2024-09-16T01:11:56.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- 172.21.15.78:0/2860431662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 msgr2=0x7f5f7010e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 --2- 172.21.15.78:0/2860431662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f7010e9a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009cd0 tx=0x7f5f6802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- 172.21.15.78:0/2860431662 shutdown_connections 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 --2- 172.21.15.78:0/2860431662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f7010e9a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 --2- 172.21.15.78:0/2860431662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 0x7f5f70101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 --2- 172.21.15.78:0/2860431662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f70107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- 172.21.15.78:0/2860431662 >> 172.21.15.78:0/2860431662 conn(0x7f5f700fd7d0 msgr2=0x7f5f700ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- 172.21.15.78:0/2860431662 shutdown_connections 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- 172.21.15.78:0/2860431662 wait complete. 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 Processor -- start 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.642+0000 7f5f785e4640 1 -- start start 2024-09-16T01:11:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 0x7f5f701983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f7019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5f70111ee0 con 0x7f5f70102470 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5f70111d60 con 0x7f5f70107b70 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f70112060 con 0x7f5f70101ad0 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54288/0 (socket says 172.21.15.78:54288) 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 -- 172.21.15.78:0/629528670 learned_addr learned my addr 172.21.15.78:0/629528670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f76380640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 0x7f5f701983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f76b81640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f7019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 -- 172.21.15.78:0/629528670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 msgr2=0x7f5f701983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 0x7f5f701983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 -- 172.21.15.78:0/629528670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 msgr2=0x7f5f7019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f7019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 -- 172.21.15.78:0/629528670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7019d3c0 con 0x7f5f70102470 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f76b81640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f7019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00d770 tx=0x7f5f6c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f637fe640 1 -- 172.21.15.78:0/629528670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c017070 con 0x7f5f70102470 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f637fe640 1 -- 172.21.15.78:0/629528670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f6c005500 con 0x7f5f70102470 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7019d6b0 con 0x7f5f70102470 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f637fe640 1 -- 172.21.15.78:0/629528670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c004e00 con 0x7f5f70102470 2024-09-16T01:11:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.643+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f701a4f40 con 0x7f5f70102470 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f637fe640 1 -- 172.21.15.78:0/629528670 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f5f6c004fa0 con 0x7f5f70102470 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 Processor -- start 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 -- start start 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a12db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a220c0 0x55bd98a244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.646+0000 7f5f7b459740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55bd989772f0 con 0x55bd98a132f0 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f7b459740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bd9891f270 con 0x55bd98a129d0 2024-09-16T01:11:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f7b459740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bd989732b0 con 0x55bd98a220c0 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54292/0 (socket says 172.21.15.78:54292) 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 -- 172.21.15.78:0/2580374600 learned_addr learned my addr 172.21.15.78:0/2580374600 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f637fe640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 0x7f5f5c0a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a220c0 0x55bd98a244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76b81640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a12db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76b81640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 0x7f5f5c0a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 -- 172.21.15.78:0/2580374600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a220c0 msgr2=0x55bd98a244b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a220c0 0x55bd98a244b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 -- 172.21.15.78:0/2580374600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 msgr2=0x55bd98a12db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a12db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f637fe640 1 -- 172.21.15.78:0/629528670 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f6c0cfd90 con 0x7f5f70102470 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 -- 172.21.15.78:0/2580374600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9891e340 con 0x55bd98a132f0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76b81640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a12db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76380640 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009ca0 tx=0x7f5f64011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589aac2442cdd189 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f60bfe640 1 -- 172.21.15.78:0/2580374600 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64004250 con 0x55bd98a132f0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f60bfe640 1 -- 172.21.15.78:0/2580374600 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f640043f0 con 0x55bd98a132f0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f60bfe640 1 -- 172.21.15.78:0/2580374600 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64019870 con 0x55bd98a132f0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.647+0000 7f5f76b81640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 0x7f5f5c0a5b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5f68004af0 tx=0x7f5f6803f040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 -- 172.21.15.78:0/2580374600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 msgr2=0x55bd98a21b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009ca0 tx=0x7f5f64011810 comp rx=0 tx=0).stop 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 -- 172.21.15.78:0/2580374600 shutdown_connections 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a220c0 0x55bd98a244b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a21b80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2580374600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a12db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 -- 172.21.15.78:0/2580374600 >> 172.21.15.78:0/2580374600 conn(0x55bd98a0f060 msgr2=0x55bd98a0f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 -- 172.21.15.78:0/2580374600 shutdown_connections 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.648+0000 7f5f7b459740 1 -- 172.21.15.78:0/2580374600 wait complete. 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 -- start start 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 Processor -- start 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a1ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a132f0 0x55bd98a1f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55bd98a24cf0 con 0x55bd98a220c0 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bd98900290 con 0x55bd98a132f0 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bd98a24e70 con 0x55bd98a129d0 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f7b459740 10 client.0 _ll_register_callbacks cb 0x55bd98931140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54306/0 (socket says 172.21.15.78:54306) 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 -- 172.21.15.78:0/2727500892 learned_addr learned my addr 172.21.15.78:0/2727500892 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76380640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a1ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a132f0 0x55bd98a1f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 -- 172.21.15.78:0/2727500892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 msgr2=0x55bd98a1ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a1ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 -- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a132f0 msgr2=0x55bd98a1f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a132f0 0x55bd98a1f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76b81640 1 -- 172.21.15.78:0/2727500892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9868a210 con 0x55bd98a220c0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.649+0000 7f5f76380640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a1ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f76b81640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5f68003bf0 tx=0x7f5f680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7d36a2f9288a33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f680047c0 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f680305a0 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f640043f0 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f680307a0 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 10 client.15543 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd98756620 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f640043f0 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5f68030070 con 0x55bd98a220c0 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 10 client.15543 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 20 client.15543 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a34510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 4 client.15543 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:56.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.650+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd98984a20 con 0x55bd98a320e0 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.651+0000 7f5f76380640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a34510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.651+0000 7f5f59ffb640 10 client.15543 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.651+0000 7f5f76380640 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a34510 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=493e70859fa228d1 server_cookie=4f79f5f3e21e4ca7 in_seq=0 out_seq=0 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f59ffb640 1 -- 172.21.15.78:0/2727500892 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd98984a20 con 0x55bd98a320e0 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f59ffb640 10 client.15543 handle_command_reply: tid=0 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 client.15543 shutdown 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 msgr2=0x55bd98a1fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5f68003bf0 tx=0x7f5f680095c0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 shutdown_connections 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a34510 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a1fa10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f59ffb640 0 client.15543 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a132f0 0x55bd98a1f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 --2- 172.21.15.78:0/2727500892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a1ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 >> 172.21.15.78:0/2727500892 conn(0x55bd98a0f060 msgr2=0x55bd98a14d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 shutdown_connections 2024-09-16T01:11:56.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 1 -- 172.21.15.78:0/2727500892 wait complete. 2024-09-16T01:11:56.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.652+0000 7f5f7b459740 20 client.15543 trim_cache size 0 max 16384 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.681+0000 7f5f7b459740 1 Processor -- start 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.681+0000 7f5f7b459740 1 -- start start 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a39e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a132f0 0x55bd98a3a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701af5e0 con 0x55bd98a220c0 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701ace80 con 0x55bd98a129d0 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f7b459740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701aacd0 con 0x55bd98a132f0 2024-09-16T01:11:56.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54322/0 (socket says 172.21.15.78:54322) 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 -- 172.21.15.78:0/1665781336 learned_addr learned my addr 172.21.15.78:0/1665781336 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76380640 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a39e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 -- 172.21.15.78:0/1665781336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a132f0 msgr2=0x55bd98a3a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a132f0 0x55bd98a3a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 -- 172.21.15.78:0/1665781336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 msgr2=0x55bd98a39e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a39e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 -- 172.21.15.78:0/1665781336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd98a13120 con 0x55bd98a220c0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f76b81640 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5f68004a80 tx=0x7f5f68031e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f30bd2a54e4c59d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1665781336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68062020 con 0x55bd98a220c0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1665781336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f68036040 con 0x55bd98a220c0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.682+0000 7f5f5bfff640 1 -- 172.21.15.78:0/1665781336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68030e00 con 0x55bd98a220c0 2024-09-16T01:11:56.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- 172.21.15.78:0/1665781336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 msgr2=0x55bd98a0d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5f68004a80 tx=0x7f5f68031e10 comp rx=0 tx=0).stop 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- 172.21.15.78:0/1665781336 shutdown_connections 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a220c0 0x55bd98a0d9a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a132f0 0x55bd98a3a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 --2- 172.21.15.78:0/1665781336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a129d0 0x55bd98a39e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- 172.21.15.78:0/1665781336 >> 172.21.15.78:0/1665781336 conn(0x55bd98a0f060 msgr2=0x55bd98a0f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- 172.21.15.78:0/1665781336 shutdown_connections 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- 172.21.15.78:0/1665781336 wait complete. 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 -- start start 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.683+0000 7f5f7b459740 1 Processor -- start 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a2a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 0x55bd98a1b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701ab340 con 0x55bd98a132f0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701ac7b0 con 0x55bd98a220c0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f701aba10 con 0x55bd98a129d0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 10 client.0 _ll_register_callbacks cb 0x7f5f64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54338/0 (socket says 172.21.15.78:54338) 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 -- 172.21.15.78:0/3400631117 learned_addr learned my addr 172.21.15.78:0/3400631117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f76b81640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 0x55bd98a1b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f76380640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a2a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 -- 172.21.15.78:0/3400631117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 msgr2=0x55bd98a2a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a2a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 -- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 msgr2=0x55bd98a1b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 0x55bd98a1b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 -- 172.21.15.78:0/3400631117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd98a1bec0 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f76b81640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 0x55bd98a1b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f75b7f640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009610 tx=0x7f5f6c0048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b1eeffe8bf2e9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0b3020 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5f6c098110 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f68036040 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f6c020040 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 10 client.15549 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.684+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd98a9c800 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f68036040 con 0x55bd98a132f0 2024-09-16T01:11:56.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5f6c0860b0 con 0x55bd98a132f0 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 10 client.15549 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 10 client.15549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 20 client.15549 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a9ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 4 client.15549 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd9897ed70 con 0x55bd98a320e0 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f76380640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a9ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f59ffb640 10 client.15549 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:56.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.685+0000 7f5f76380640 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a9ef60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1bcdb6824e0fb47 server_cookie=212f6a375b0f7e3c in_seq=0 out_seq=0 2024-09-16T01:11:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f59ffb640 1 -- 172.21.15.78:0/3400631117 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd9897ed70 con 0x55bd98a320e0 2024-09-16T01:11:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f59ffb640 10 client.15549 handle_command_reply: tid=0 2024-09-16T01:11:56.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 client.15549 shutdown 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 msgr2=0x55bd98a2a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c009610 tx=0x7f5f6c0048a0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 shutdown_connections 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55bd98a320e0 0x55bd98a9ef60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.689+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bd98a220c0 0x55bd98a1b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd98a132f0 0x55bd98a2a7f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 1 --2- 172.21.15.78:0/3400631117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bd98a129d0 0x55bd98a2a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f59ffb640 0 client.15549 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 >> 172.21.15.78:0/3400631117 conn(0x55bd98a0f060 msgr2=0x55bd98a207f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 shutdown_connections 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 1 -- 172.21.15.78:0/3400631117 wait complete. 2024-09-16T01:11:56.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f7b459740 20 client.15549 trim_cache size 0 max 16384 2024-09-16T01:11:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:11:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:11:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:11:56.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 msgr2=0x7f5f5c0a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 0x7f5f5c0a5b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5f68004af0 tx=0x7f5f6803f040 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 msgr2=0x7f5f70198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00d770 tx=0x7f5f6c00dc30 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 shutdown_connections 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5f5c0a3690 0x7f5f5c0a5b50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5f70107b70 0x7f5f7019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f70102470 0x7f5f70198910 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 --2- 172.21.15.78:0/629528670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f70101ad0 0x7f5f701983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 >> 172.21.15.78:0/629528670 conn(0x7f5f700fd7d0 msgr2=0x7f5f700fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 shutdown_connections 2024-09-16T01:11:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:11:56.690+0000 7f5f785e4640 1 -- 172.21.15.78:0/629528670 wait complete. 2024-09-16T01:11:56.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:56 smithi078 ceph-mon[30937]: pgmap v37: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.783+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2078109010 con 0x7f2078100a80 2024-09-16T01:11:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.783+0000 7f206effd640 1 -- 172.21.15.78:0/3524059201 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11922 (secure 0 0 0) 0x7f207409c490 con 0x7f2078100a80 2024-09-16T01:11:56.786 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:11:56.786 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":28,"btime":"2024-09-16T01:11:13:114118+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:11:12.780839+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:11:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 28 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 msgr2=0x7f20640a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 0x7f20640a5a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20680096c0 tx=0x7f2068000bc0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 msgr2=0x7f2078199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f207400c890 tx=0x7f207400cd50 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 shutdown_connections 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f20640a3560 0x7f20640a5a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2078106a80 0x7f20781a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2078101420 0x7f2078199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 --2- 172.21.15.78:0/3524059201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2078100a80 0x7f2078199330 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 >> 172.21.15.78:0/3524059201 conn(0x7f20780fc7a0 msgr2=0x7f20780fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 shutdown_connections 2024-09-16T01:11:56.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:56.786+0000 7f207fe97640 1 -- 172.21.15.78:0/3524059201 wait complete. 2024-09-16T01:11:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:56 smithi193 ceph-mon[35701]: pgmap v37: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:56.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:56 smithi148 ceph-mon[35030]: pgmap v37: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:57.556 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 538a64bd --release --await' 2024-09-16T01:11:57.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 538a64bd --release --await 2024-09-16T01:11:57.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:57 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3524059201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:57.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:57 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3524059201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:57.853 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:11:57.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:57 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3524059201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:11:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.432+0000 7fc851814640 1 -- 172.21.15.78:0/43202506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c102d30 msgr2=0x7fc84c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.432+0000 7fc851814640 1 --2- 172.21.15.78:0/43202506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c102d30 0x7fc84c103110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009cd0 tx=0x7fc83c02a910 comp rx=0 tx=0).stop 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 -- 172.21.15.78:0/43202506 shutdown_connections 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 --2- 172.21.15.78:0/43202506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 0x7fc84c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 --2- 172.21.15.78:0/43202506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c1036e0 0x7fc84c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 --2- 172.21.15.78:0/43202506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c102d30 0x7fc84c103110 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 -- 172.21.15.78:0/43202506 >> 172.21.15.78:0/43202506 conn(0x7fc84c0fc7d0 msgr2=0x7fc84c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 -- 172.21.15.78:0/43202506 shutdown_connections 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 -- 172.21.15.78:0/43202506 wait complete. 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 Processor -- start 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 -- start start 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.433+0000 7fc851814640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c102d30 0x7fc84c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 0x7fc84c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c112160 con 0x7fc84c1036e0 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c111fe0 con 0x7fc84c1082a0 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1122e0 con 0x7fc84c102d30 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54362/0 (socket says 172.21.15.78:54362) 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 -- 172.21.15.78:0/3528087288 learned_addr learned my addr 172.21.15.78:0/3528087288 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84affd640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c102d30 0x7fc84c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 0x7fc84c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 -- 172.21.15.78:0/3528087288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c102d30 msgr2=0x7fc84c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c102d30 0x7fc84c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 -- 172.21.15.78:0/3528087288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 msgr2=0x7fc84c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 0x7fc84c19d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 -- 172.21.15.78:0/3528087288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c19d6f0 con 0x7fc84c1036e0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc834006fd0 tx=0x7fc83400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc850812640 1 -- 172.21.15.78:0/3528087288 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834011830 con 0x7fc84c1036e0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c19d9e0 con 0x7fc84c1036e0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc850812640 1 -- 172.21.15.78:0/3528087288 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc834016070 con 0x7fc84c1036e0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc850812640 1 -- 172.21.15.78:0/3528087288 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83400f2e0 con 0x7fc84c1036e0 2024-09-16T01:11:58.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.434+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1a52c0 con 0x7fc84c1036e0 2024-09-16T01:11:58.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 Processor -- start 2024-09-16T01:11:58.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 -- start start 2024-09-16T01:11:58.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc850812640 1 -- 172.21.15.78:0/3528087288 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7fc83400f480 con 0x7fc84c1036e0 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be01440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be0bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55596bd66620 con 0x55596be0c240 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55596bd0de50 con 0x55596be01980 2024-09-16T01:11:58.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.437+0000 7fc853bed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55596bd625e0 con 0x55596be01060 2024-09-16T01:11:58.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54378/0 (socket says 172.21.15.78:54378) 2024-09-16T01:11:58.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1227217026 learned_addr learned my addr 172.21.15.78:0/1227217026 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:58.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be01440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84affd640 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be0bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1227217026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 msgr2=0x55596be01440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be01440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1227217026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 msgr2=0x55596be0bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be0bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1227217026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55596bcfed40 con 0x55596be0c240 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc8340ccc10 tx=0x7fc8340d5ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454f9c914622c29b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc850812640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 0x7fc8380a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc841ffb640 1 -- 172.21.15.78:0/1227217026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340de070 con 0x55596be0c240 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc841ffb640 1 -- 172.21.15.78:0/1227217026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83409bf00 con 0x55596be0c240 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc841ffb640 1 -- 172.21.15.78:0/1227217026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83406cc80 con 0x55596be0c240 2024-09-16T01:11:58.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.438+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 0x7fc8380a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc850812640 1 -- 172.21.15.78:0/3528087288 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc8340cf9e0 con 0x7fc84c1036e0 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 -- 172.21.15.78:0/1227217026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 msgr2=0x55596be0e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 0x7fc8380a5a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc84400a9d0 tx=0x7fc844008040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc8340ccc10 tx=0x7fc8340d5ef0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 -- 172.21.15.78:0/1227217026 shutdown_connections 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be0c240 0x55596be0e630 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be0bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 --2- 172.21.15.78:0/1227217026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be01440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 -- 172.21.15.78:0/1227217026 >> 172.21.15.78:0/1227217026 conn(0x55596bdfd5f0 msgr2=0x55596bdfd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 -- 172.21.15.78:0/1227217026 shutdown_connections 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.439+0000 7fc853bed740 1 -- 172.21.15.78:0/1227217026 wait complete. 2024-09-16T01:11:58.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 -- start start 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 Processor -- start 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01060 0x55596be09090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be09b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55596be0ee10 con 0x55596be01980 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55596bceef10 con 0x55596be01060 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55596be0ef90 con 0x55596be0c240 2024-09-16T01:11:58.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc853bed740 10 client.0 _ll_register_callbacks cb 0x55596bd12db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54382/0 (socket says 172.21.15.78:54382) 2024-09-16T01:11:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1622393755 learned_addr learned my addr 172.21.15.78:0/1622393755 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be09b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84affd640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01060 0x55596be09090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1622393755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 msgr2=0x55596be09b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.440+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be09b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01060 msgr2=0x55596be09090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01060 0x55596be09090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc84a7fc640 1 -- 172.21.15.78:0/1622393755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55596ba75580 con 0x55596be01980 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc834097c60 tx=0x7fc834010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a14b4fb4427c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83406b810 con 0x55596be01980 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc834097e70 con 0x55596be01980 2024-09-16T01:11:58.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc83409bf00 con 0x55596be01980 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc8340de070 con 0x55596be01980 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 10 client.15561 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55596bb3f960 con 0x55596be01980 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83409bf00 con 0x55596be01980 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fc8340d9040 con 0x55596be01980 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 10 client.15561 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 10 client.15561 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 20 client.15561 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.441+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be1bba0 0x55596be1df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.442+0000 7fc853bed740 4 client.15561 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.442+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55596bd73b90 con 0x55596be1bba0 2024-09-16T01:11:58.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.442+0000 7fc84affd640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be1bba0 0x55596be1df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.442+0000 7fc82affd640 10 client.15561 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.442+0000 7fc84affd640 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be1bba0 0x55596be1df90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb393a24677f443 server_cookie=3a9d7bb787d941c in_seq=0 out_seq=0 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc82affd640 1 -- 172.21.15.78:0/1622393755 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55596bd73b90 con 0x55596be1bba0 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc82affd640 10 client.15561 handle_command_reply: tid=0 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 client.15561 shutdown 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 msgr2=0x55596be095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc834097c60 tx=0x7fc834010740 comp rx=0 tx=0).stop 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 shutdown_connections 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be1bba0 0x55596be1df90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be09b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be095d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 --2- 172.21.15.78:0/1622393755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01060 0x55596be09090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc82affd640 0 client.15561 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:58.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 >> 172.21.15.78:0/1622393755 conn(0x55596bdfd5f0 msgr2=0x55596be032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 shutdown_connections 2024-09-16T01:11:58.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 1 -- 172.21.15.78:0/1622393755 wait complete. 2024-09-16T01:11:58.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.443+0000 7fc853bed740 20 client.15561 trim_cache size 0 max 16384 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 Processor -- start 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 -- start start 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be20bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 0x55596be06310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1b19e0 con 0x55596be01980 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1af280 con 0x55596be0c240 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc853bed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad0d0 con 0x55596be01060 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.460+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54386/0 (socket says 172.21.15.78:54386) 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 -- 172.21.15.78:0/849351885 learned_addr learned my addr 172.21.15.78:0/849351885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84affd640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be20bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 -- 172.21.15.78:0/849351885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 msgr2=0x55596be20bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be20bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 -- 172.21.15.78:0/849351885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 msgr2=0x55596be06310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 0x55596be06310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 0x55596be06310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 -- 172.21.15.78:0/849351885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55596bdfd420 con 0x55596be01980 2024-09-16T01:11:58.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 0x55596be06310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:11:58.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc83400bde0 tx=0x7fc83400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d55c351bdbf8693 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc8417fa640 1 -- 172.21.15.78:0/849351885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83406ba00 con 0x55596be01980 2024-09-16T01:11:58.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc8417fa640 1 -- 172.21.15.78:0/849351885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8340d5040 con 0x55596be01980 2024-09-16T01:11:58.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.461+0000 7fc8417fa640 1 -- 172.21.15.78:0/849351885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83406bbe0 con 0x55596be01980 2024-09-16T01:11:58.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- 172.21.15.78:0/849351885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 msgr2=0x55596be21110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc83400bde0 tx=0x7fc83400bf90 comp rx=0 tx=0).stop 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- 172.21.15.78:0/849351885 shutdown_connections 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be0c240 0x55596be06310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01980 0x55596be21110 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 --2- 172.21.15.78:0/849351885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be01060 0x55596be20bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- 172.21.15.78:0/849351885 >> 172.21.15.78:0/849351885 conn(0x55596bdfd5f0 msgr2=0x55596be07960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- 172.21.15.78:0/849351885 shutdown_connections 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- 172.21.15.78:0/849351885 wait complete. 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.462+0000 7fc853bed740 1 -- start start 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 Processor -- start 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be21300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be1f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad740 con 0x55596be01060 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1aebb0 con 0x55596be01980 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ade10 con 0x55596be0c240 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc853bed740 10 client.0 _ll_register_callbacks cb 0x7fc8340d58e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54400/0 (socket says 172.21.15.78:54400) 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 -- 172.21.15.78:0/930175155 learned_addr learned my addr 172.21.15.78:0/930175155 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84b7fe640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be1f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 -- 172.21.15.78:0/930175155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 msgr2=0x55596be1f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be21300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be1f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 -- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 msgr2=0x55596be21300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be21300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 -- 172.21.15.78:0/930175155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55596be03ed0 con 0x55596be01060 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc84affd640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc83c02ae30 tx=0x7fc83c0098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68bc1564857eef1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.463+0000 7fc82affd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c031d90 con 0x55596be01060 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fc83c003970 con 0x55596be01060 2024-09-16T01:11:58.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8340d5040 con 0x55596be01060 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc83c003b70 con 0x55596be01060 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 10 client.15567 fetch_fsmap learned FSMap version 28 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55596be86e70 con 0x55596be01060 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8340d5040 con 0x55596be01060 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fc83c0043c0 con 0x55596be01060 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 10 client.15567 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 10 client.15567 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 20 client.15567 populate_metadata read hostname 'smithi078' 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be87620 0x55596be89a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 4 client.15567 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "538a64bd", "release": true, "await": true}] 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "538a64bd", "release": true, "await": true}) -- 0x55596bd6e0a0 con 0x55596be87620 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.464+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be87620 0x55596be89a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:11:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.465+0000 7fc82affd640 10 client.15567 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:58.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.465+0000 7fc84a7fc640 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be87620 0x55596be89a40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d97dd0fec054564b server_cookie=3212d1ec9add6a86 in_seq=0 out_seq=0 2024-09-16T01:11:58.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc82affd640 1 -- 172.21.15.78:0/930175155 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55596bd6e0a0 con 0x55596be87620 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc82affd640 10 client.15567 handle_command_reply: tid=0 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 client.15567 shutdown 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 msgr2=0x55596be20dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc83c02ae30 tx=0x7fc83c0098b0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 shutdown_connections 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55596be87620 0x55596be89a40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55596be0c240 0x55596be1f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55596be01980 0x55596be21300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 --2- 172.21.15.78:0/930175155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55596be01060 0x55596be20dc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 >> 172.21.15.78:0/930175155 conn(0x55596bdfd5f0 msgr2=0x55596be08fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc82affd640 0 client.15567 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 shutdown_connections 2024-09-16T01:11:58.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 1 -- 172.21.15.78:0/930175155 wait complete. 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.539+0000 7fc853bed740 20 client.15567 trim_cache size 0 max 16384 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 28, 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 4, 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "538a64bd": { 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 4, 2024-09-16T01:11:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 150.2, 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:11:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:11:58.545 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 msgr2=0x7fc8380a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 0x7fc8380a5a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc84400a9d0 tx=0x7fc844008040 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 msgr2=0x7fc84c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc834006fd0 tx=0x7fc83400b8e0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 shutdown_connections 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc8380a3560 0x7fc8380a5a20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc84c1082a0 0x7fc84c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc84c1036e0 0x7fc84c198c80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 --2- 172.21.15.78:0/3528087288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc84c102d30 0x7fc84c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 >> 172.21.15.78:0/3528087288 conn(0x7fc84c0fc7d0 msgr2=0x7fc84c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:11:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 shutdown_connections 2024-09-16T01:11:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:11:58.540+0000 7fc851814640 1 -- 172.21.15.78:0/3528087288 wait complete. 2024-09-16T01:11:58.781 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:11:58 smithi078 ceph-mon[30937]: pgmap v38: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:58.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:11:58 smithi193 ceph-mon[35701]: pgmap v38: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:58.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:11:58 smithi148 ceph-mon[35030]: pgmap v38: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 232 op/s 2024-09-16T01:11:59.302 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '538a64bd', total seconds elapsed: 33.8 2024-09-16T01:12:00.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:00 smithi078 ceph-mon[30937]: pgmap v39: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:12:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:00 smithi193 ceph-mon[35701]: pgmap v39: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:12:00.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:00 smithi148 ceph-mon[35030]: pgmap v39: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 25 MiB/s wr, 235 op/s 2024-09-16T01:12:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 45.0 s, 1318.4 tps, lat 487.974 ms stddev 3641.774 2024-09-16T01:12:02.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:02 smithi078 ceph-mon[30937]: pgmap v40: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9.9 MiB/s wr, 94 op/s 2024-09-16T01:12:02.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:02 smithi193 ceph-mon[35701]: pgmap v40: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9.9 MiB/s wr, 94 op/s 2024-09-16T01:12:02.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:02 smithi148 ceph-mon[35030]: pgmap v40: 129 pgs: 129 active+clean; 950 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9.9 MiB/s wr, 94 op/s 2024-09-16T01:12:04.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:04 smithi078 ceph-mon[30937]: pgmap v41: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 731 op/s 2024-09-16T01:12:04.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:04 smithi193 ceph-mon[35701]: pgmap v41: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 731 op/s 2024-09-16T01:12:04.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:04 smithi148 ceph-mon[35030]: pgmap v41: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 731 op/s 2024-09-16T01:12:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 50.0 s, 2692.4 tps, lat 37.181 ms stddev 43.725 2024-09-16T01:12:06.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:06 smithi078 ceph-mon[30937]: pgmap v42: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:06.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:06 smithi193 ceph-mon[35701]: pgmap v42: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:06.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:06 smithi148 ceph-mon[35030]: pgmap v42: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:08.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:08 smithi078 ceph-mon[30937]: pgmap v43: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:08.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:08 smithi193 ceph-mon[35701]: pgmap v43: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:08.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:08 smithi148 ceph-mon[35030]: pgmap v43: 129 pgs: 129 active+clean; 964 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 -- 172.21.15.78:0/2233267722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8105ea0 msgr2=0x7f14d8106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 --2- 172.21.15.78:0/2233267722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8105ea0 0x7f14d8106310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009ca0 tx=0x7f14d002a970 comp rx=0 tx=0).stop 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 -- 172.21.15.78:0/2233267722 shutdown_connections 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 --2- 172.21.15.78:0/2233267722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8105ea0 0x7f14d8106310 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 --2- 172.21.15.78:0/2233267722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 0x7f14d8105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 --2- 172.21.15.78:0/2233267722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8100a70 0x7f14d8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.821+0000 7f14dd731640 1 -- 172.21.15.78:0/2233267722 >> 172.21.15.78:0/2233267722 conn(0x7f14d8077990 msgr2=0x7f14d80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.822+0000 7f14dd731640 1 -- 172.21.15.78:0/2233267722 shutdown_connections 2024-09-16T01:12:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.823+0000 7f14dd731640 1 -- 172.21.15.78:0/2233267722 wait complete. 2024-09-16T01:12:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 Processor -- start 2024-09-16T01:12:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 -- start start 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8100a70 0x7f14d8114db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 0x7f14d810ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f14d8107870 con 0x7f14d8101390 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81076f0 con 0x7f14d8105ea0 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.824+0000 7f14dd731640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81079f0 con 0x7f14d8100a70 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.825+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.825+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59392/0 (socket says 172.21.15.78:59392) 2024-09-16T01:12:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.825+0000 7f14d77fe640 1 -- 172.21.15.78:0/157893034 learned_addr learned my addr 172.21.15.78:0/157893034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.825+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8100a70 0x7f14d8114db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.825+0000 7f14d67fc640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 0x7f14d810ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 -- 172.21.15.78:0/157893034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8100a70 msgr2=0x7f14d8114db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8100a70 0x7f14d8114db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 -- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 msgr2=0x7f14d810ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 0x7f14d810ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 -- 172.21.15.78:0/157893034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d810fab0 con 0x7f14d8105ea0 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.826+0000 7f14d77fe640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14d0005e30 tx=0x7f14d0003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.827+0000 7f14bffff640 1 -- 172.21.15.78:0/157893034 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0047070 con 0x7f14d8105ea0 2024-09-16T01:12:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.827+0000 7f14bffff640 1 -- 172.21.15.78:0/157893034 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d0030b00 con 0x7f14d8105ea0 2024-09-16T01:12:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.827+0000 7f14bffff640 1 -- 172.21.15.78:0/157893034 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0034070 con 0x7f14d8105ea0 2024-09-16T01:12:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.827+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d81b3820 con 0x7f14d8105ea0 2024-09-16T01:12:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.828+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d81b3bd0 con 0x7f14d8105ea0 2024-09-16T01:12:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.830+0000 7f14bffff640 1 -- 172.21.15.78:0/157893034 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 25) ==== 138639+0+0 (secure 0 0 0) 0x7f14d0037020 con 0x7f14d8105ea0 2024-09-16T01:12:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 Processor -- start 2024-09-16T01:12:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 -- start start 2024-09-16T01:12:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5649c32fa2f0 con 0x5649c3396410 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5649c32a2270 con 0x5649c3395af0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14e04d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649c32f62b0 con 0x5649c33a0cb0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14bffff640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 0x7f14c00a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14d77fe640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 0x7f14c00a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59398/0 (socket says 172.21.15.78:59398) 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14d67fc640 1 -- 172.21.15.78:0/262535986 learned_addr learned my addr 172.21.15.78:0/262535986 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.831+0000 7f14d67fc640 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14bffff640 1 -- 172.21.15.78:0/157893034 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f14d00f0d00 con 0x7f14d8105ea0 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d67fc640 1 -- 172.21.15.78:0/262535986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 msgr2=0x5649c33a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d67fc640 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d67fc640 1 -- 172.21.15.78:0/262535986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 msgr2=0x5649c33a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d67fc640 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d67fc640 1 -- 172.21.15.78:0/262535986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c32a1340 con 0x5649c3395af0 2024-09-16T01:12:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.832+0000 7f14d77fe640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 0x7f14c00a5c70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14d00edee0 tx=0x7f14d00a7040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.833+0000 7f14d67fc640 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009560 tx=0x7f14c8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c88a06329cb9ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.834+0000 7f14bd7fa640 1 -- 172.21.15.78:0/262535986 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8012ca0 con 0x5649c3395af0 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.834+0000 7f14bd7fa640 1 -- 172.21.15.78:0/262535986 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c8012e40 con 0x5649c3395af0 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.834+0000 7f14bd7fa640 1 -- 172.21.15.78:0/262535986 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8003850 con 0x5649c3395af0 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 -- 172.21.15.78:0/262535986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 msgr2=0x5649c3395ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009560 tx=0x7f14c8009780 comp rx=0 tx=0).stop 2024-09-16T01:12:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 -- 172.21.15.78:0/262535986 shutdown_connections 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 --2- 172.21.15.78:0/262535986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c3395ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 -- 172.21.15.78:0/262535986 >> 172.21.15.78:0/262535986 conn(0x5649c3392060 msgr2=0x5649c3392440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 -- 172.21.15.78:0/262535986 shutdown_connections 2024-09-16T01:12:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.835+0000 7f14e04d0740 1 -- 172.21.15.78:0/262535986 wait complete. 2024-09-16T01:12:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 -- start start 2024-09-16T01:12:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 Processor -- start 2024-09-16T01:12:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 0x5649c339dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c339e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 0x5649c339e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5649c33a3880 con 0x5649c3395af0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5649c3283290 con 0x5649c33a0cb0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649c33a3a00 con 0x5649c3396410 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14e04d0740 10 client.0 _ll_register_callbacks cb 0x5649c32b4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 0x5649c339dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 0x5649c339dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53756/0 (socket says 172.21.15.78:53756) 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.836+0000 7f14d67fc640 1 -- 172.21.15.78:0/3722163651 learned_addr learned my addr 172.21.15.78:0/3722163651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 0x5649c339e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c339e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 -- 172.21.15.78:0/3722163651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 msgr2=0x5649c339e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c339e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 -- 172.21.15.78:0/3722163651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 msgr2=0x5649c339dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 0x5649c339dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 -- 172.21.15.78:0/3722163651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c300d210 con 0x5649c33a0cb0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c339e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.837+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 0x5649c339e6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f14c800a010 tx=0x7f14c8010ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b2dd4d62f2a5b92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14baff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8005020 con 0x5649c33a0cb0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649c3307a20 con 0x5649c33a0cb0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f14c80051c0 con 0x5649c33a0cb0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14c8033050 con 0x5649c33a0cb0 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14e04d0740 10 client.25070 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:09.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.838+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649c30d9620 con 0x5649c33a0cb0 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5649c3307a20 con 0x5649c33a0cb0 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f14c80048c0 con 0x5649c33a0cb0 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 10 client.25070 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 10 client.25070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 20 client.25070 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c33b0fd0 0x5649c33b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 4 client.25070 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649c3301d70 con 0x5649c33b0fd0 2024-09-16T01:12:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.839+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c33b0fd0 0x5649c33b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.840+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c33b0fd0 0x5649c33b33f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99e9e26e2c7c68dd server_cookie=1d9bfc496130e9c5 in_seq=0 out_seq=0 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.840+0000 7f14baff5640 10 client.25070 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14baff5640 1 -- 172.21.15.78:0/3722163651 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649c3301d70 con 0x5649c33b0fd0 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14baff5640 10 client.25070 handle_command_reply: tid=0 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 client.25070 shutdown 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 msgr2=0x5649c339e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 0x5649c339e6e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f14c800a010 tx=0x7f14c8010ca0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 shutdown_connections 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c33b0fd0 0x5649c33b33f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c33a0cb0 0x5649c339e6e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c339e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3722163651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3395af0 0x5649c339dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 >> 172.21.15.78:0/3722163651 conn(0x5649c3392060 msgr2=0x5649c3397dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14baff5640 0 client.25070 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 shutdown_connections 2024-09-16T01:12:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 1 -- 172.21.15.78:0/3722163651 wait complete. 2024-09-16T01:12:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.841+0000 7f14e04d0740 20 client.25070 trim_cache size 0 max 16384 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 Processor -- start 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 -- start start 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c33b8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c33a0cb0 0x5649c33b9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81c05c0 con 0x5649c33a0cb0 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81bde60 con 0x5649c3395af0 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14e04d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81bbcb0 con 0x5649c3396410 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59416/0 (socket says 172.21.15.78:59416) 2024-09-16T01:12:09.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14d67fc640 1 -- 172.21.15.78:0/3622399162 learned_addr learned my addr 172.21.15.78:0/3622399162 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.863+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c33a0cb0 0x5649c33b9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 -- 172.21.15.78:0/3622399162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 msgr2=0x5649c33b8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c33b8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 -- 172.21.15.78:0/3622399162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c33a0cb0 msgr2=0x5649c33b9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c33a0cb0 0x5649c33b9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 -- 172.21.15.78:0/3622399162 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c339ea50 con 0x5649c3395af0 2024-09-16T01:12:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.864+0000 7f14d67fc640 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f14c8006010 tx=0x7f14c8010ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98fb713663a005b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.865+0000 7f14bcff9640 1 -- 172.21.15.78:0/3622399162 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8009630 con 0x5649c3395af0 2024-09-16T01:12:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.865+0000 7f14bcff9640 1 -- 172.21.15.78:0/3622399162 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c80097d0 con 0x5649c3395af0 2024-09-16T01:12:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.865+0000 7f14bcff9640 1 -- 172.21.15.78:0/3622399162 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c80050c0 con 0x5649c3395af0 2024-09-16T01:12:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- 172.21.15.78:0/3622399162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 msgr2=0x5649c339e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f14c8006010 tx=0x7f14c8010ca0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- 172.21.15.78:0/3622399162 shutdown_connections 2024-09-16T01:12:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c33a0cb0 0x5649c33b9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3396410 0x5649c33b8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 --2- 172.21.15.78:0/3622399162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c339e510 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- 172.21.15.78:0/3622399162 >> 172.21.15.78:0/3622399162 conn(0x5649c3392060 msgr2=0x5649c3398ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- 172.21.15.78:0/3622399162 shutdown_connections 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- 172.21.15.78:0/3622399162 wait complete. 2024-09-16T01:12:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.866+0000 7f14e04d0740 1 -- start start 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 Processor -- start 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c33a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81bc320 con 0x5649c3396410 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81bd790 con 0x5649c3395af0 2024-09-16T01:12:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81bc9f0 con 0x5649c33a0cb0 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14e04d0740 10 client.0 _ll_register_callbacks cb 0x7f14cc00b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53766/0 (socket says 172.21.15.78:53766) 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 -- 172.21.15.78:0/1931891491 learned_addr learned my addr 172.21.15.78:0/1931891491 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c33a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d67fc640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 -- 172.21.15.78:0/1931891491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 msgr2=0x5649c33a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 -- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 msgr2=0x5649c33a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c33a9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d6ffd640 1 -- 172.21.15.78:0/1931891491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c33a4410 con 0x5649c3396410 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.867+0000 7f14d67fc640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c33a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14d6ffd640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5649c339d4d0 tx=0x7f14cc0157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ed64a2291e50de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14baff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc020030 con 0x5649c3396410 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f14cc00f040 con 0x5649c3396410 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14c80097d0 con 0x5649c3396410 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14cc02d050 con 0x5649c3396410 2024-09-16T01:12:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14e04d0740 10 client.15585 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.868+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649c341a9e0 con 0x5649c3396410 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c80097d0 con 0x5649c3396410 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f14cc01c250 con 0x5649c3396410 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 10 client.15585 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 20 client.15585 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c341b290 0x5649c341d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 4 client.15585 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:12:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649c327e900 con 0x5649c341b290 2024-09-16T01:12:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.869+0000 7f14d67fc640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c341b290 0x5649c341d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.870+0000 7f14baff5640 10 client.15585 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.870+0000 7f14d67fc640 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c341b290 0x5649c341d6b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5191de14b8d009b1 server_cookie=16000cef62a2a3f3 in_seq=0 out_seq=0 2024-09-16T01:12:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.878+0000 7f14baff5640 1 -- 172.21.15.78:0/1931891491 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649c327e900 con 0x5649c341b290 2024-09-16T01:12:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.878+0000 7f14baff5640 10 client.15585 handle_command_reply: tid=0 2024-09-16T01:12:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.878+0000 7f14e04d0740 1 client.15585 shutdown 2024-09-16T01:12:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.879+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 msgr2=0x5649c33a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.879+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5649c339d4d0 tx=0x7f14cc0157f0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.879+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 shutdown_connections 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.879+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5649c341b290 0x5649c341d6b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c33a0cb0 0x5649c33a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14baff5640 0 client.15585 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649c3396410 0x5649c33a3990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 --2- 172.21.15.78:0/1931891491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c3395af0 0x5649c33a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 >> 172.21.15.78:0/1931891491 conn(0x5649c3392060 msgr2=0x5649c339f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 shutdown_connections 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 1 -- 172.21.15.78:0/1931891491 wait complete. 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.880+0000 7f14e04d0740 20 client.15585 trim_cache size 0 max 16384 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:12:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:12:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:12:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 msgr2=0x7f14c00a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 0x7f14c00a5c70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f14d00edee0 tx=0x7f14d00a7040 comp rx=0 tx=0).stop 2024-09-16T01:12:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 msgr2=0x7f14d810f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14d0005e30 tx=0x7f14d0003e80 comp rx=0 tx=0).stop 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 shutdown_connections 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f14c00a37b0 0x7f14c00a5c70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14d8105ea0 0x7f14d810f3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14d8101390 0x7f14d810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 --2- 172.21.15.78:0/157893034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d8100a70 0x7f14d8114db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 >> 172.21.15.78:0/157893034 conn(0x7f14d8077990 msgr2=0x7f14d8103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 shutdown_connections 2024-09-16T01:12:09.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:09.881+0000 7f14dd731640 1 -- 172.21.15.78:0/157893034 wait complete. 2024-09-16T01:12:10.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:10 smithi078 ceph-mon[30937]: pgmap v44: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-09-16T01:12:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:10 smithi193 ceph-mon[35701]: pgmap v44: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-09-16T01:12:10.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:10 smithi148 ceph-mon[35030]: pgmap v44: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-09-16T01:12:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 55.0 s, 3421.8 tps, lat 29.252 ms stddev 32.281 2024-09-16T01:12:12.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:12 smithi078 ceph-mon[30937]: pgmap v45: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.9 MiB/s wr, 930 op/s 2024-09-16T01:12:12.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:12 smithi193 ceph-mon[35701]: pgmap v45: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.9 MiB/s wr, 930 op/s 2024-09-16T01:12:12.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:12 smithi148 ceph-mon[35030]: pgmap v45: 129 pgs: 129 active+clean; 1002 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.9 MiB/s wr, 930 op/s 2024-09-16T01:12:14.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:14 smithi078 ceph-mon[30937]: pgmap v46: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-09-16T01:12:14.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:14 smithi193 ceph-mon[35701]: pgmap v46: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-09-16T01:12:14.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:14 smithi148 ceph-mon[35030]: pgmap v46: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-09-16T01:12:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 60.0 s, 3045.4 tps, lat 32.712 ms stddev 44.015 2024-09-16T01:12:16.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:16 smithi078 ceph-mon[30937]: pgmap v47: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:16.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:16 smithi193 ceph-mon[35701]: pgmap v47: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:16 smithi148 ceph-mon[35030]: pgmap v47: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:18.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:18 smithi193 ceph-mon[35701]: pgmap v48: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:18.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:18 smithi148 ceph-mon[35030]: pgmap v48: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:18 smithi078 ceph-mon[30937]: pgmap v48: 129 pgs: 129 active+clean; 1020 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.4 MiB/s wr, 620 op/s 2024-09-16T01:12:20.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:20 smithi193 ceph-mon[35701]: pgmap v49: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 1.14k op/s 2024-09-16T01:12:20.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:20 smithi193 ceph-mon[35701]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-09-16T01:12:20.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:20 smithi148 ceph-mon[35030]: pgmap v49: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 1.14k op/s 2024-09-16T01:12:20.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:20 smithi148 ceph-mon[35030]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-09-16T01:12:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 65.0 s, 3077.2 tps, lat 32.643 ms stddev 34.481 2024-09-16T01:12:21.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:20 smithi078 ceph-mon[30937]: pgmap v49: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 16 MiB/s wr, 1.14k op/s 2024-09-16T01:12:21.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:20 smithi078 ceph-mon[30937]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-09-16T01:12:22.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:22 smithi193 ceph-mon[35701]: pgmap v50: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 845 op/s 2024-09-16T01:12:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:22 smithi148 ceph-mon[35030]: pgmap v50: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 845 op/s 2024-09-16T01:12:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.014+0000 7f5351223640 1 -- 172.21.15.78:0/820152111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c105f10 msgr2=0x7f534c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.014+0000 7f5351223640 1 --2- 172.21.15.78:0/820152111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c105f10 0x7f534c1063c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f533c01a940 tx=0x7f533c03b470 comp rx=0 tx=0).stop 2024-09-16T01:12:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 -- 172.21.15.78:0/820152111 shutdown_connections 2024-09-16T01:12:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 --2- 172.21.15.78:0/820152111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c105f10 0x7f534c1063c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 --2- 172.21.15.78:0/820152111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f534c1013b0 0x7f534c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 --2- 172.21.15.78:0/820152111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c100a90 0x7f534c100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 -- 172.21.15.78:0/820152111 >> 172.21.15.78:0/820152111 conn(0x7f534c0fc6b0 msgr2=0x7f534c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.015+0000 7f5351223640 1 -- 172.21.15.78:0/820152111 shutdown_connections 2024-09-16T01:12:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.017+0000 7f5351223640 1 -- 172.21.15.78:0/820152111 wait complete. 2024-09-16T01:12:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.017+0000 7f5351223640 1 Processor -- start 2024-09-16T01:12:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 -- start start 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c1013b0 0x7f534c110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f534c105f10 0x7f534c111290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f534c109810 con 0x7f534c100a90 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f534c109690 con 0x7f534c105f10 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.018+0000 7f5351223640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f534c109990 con 0x7f534c1013b0 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.019+0000 7f534ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.019+0000 7f534ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48212/0 (socket says 172.21.15.78:48212) 2024-09-16T01:12:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.019+0000 7f534ad9d640 1 -- 172.21.15.78:0/3323144454 learned_addr learned my addr 172.21.15.78:0/3323144454 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.019+0000 7f534a59c640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c1013b0 0x7f534c110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.019+0000 7f534ad9d640 1 -- 172.21.15.78:0/3323144454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c1013b0 msgr2=0x7f534c110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f534ad9d640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c1013b0 0x7f534c110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f534ad9d640 1 -- 172.21.15.78:0/3323144454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f534c105f10 msgr2=0x7f534c111290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:12:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f534ad9d640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f534c105f10 0x7f534c111290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f534ad9d640 1 -- 172.21.15.78:0/3323144454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c111850 con 0x7f534c100a90 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f534ad9d640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f534400ba80 tx=0x7f534400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.020+0000 7f5343fff640 1 -- 172.21.15.78:0/3323144454 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344019070 con 0x7f534c100a90 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.021+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c111b40 con 0x7f534c100a90 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.021+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c1a2c00 con 0x7f534c100a90 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.021+0000 7f5343fff640 1 -- 172.21.15.78:0/3323144454 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534400cba0 con 0x7f534c100a90 2024-09-16T01:12:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.021+0000 7f5343fff640 1 -- 172.21.15.78:0/3323144454 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344007610 con 0x7f534c100a90 2024-09-16T01:12:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.024+0000 7f5353fbe740 1 Processor -- start 2024-09-16T01:12:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.024+0000 7f5353fbe740 1 -- start start 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x5632925520a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5632924a92f0 con 0x56329254fcb0 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563292451270 con 0x563292545410 2024-09-16T01:12:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f5353fbe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5632924a52b0 con 0x563292544af0 2024-09-16T01:12:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f534b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f534b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:60278/0 (socket says 172.21.15.78:60278) 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.025+0000 7f534b59e640 1 -- 172.21.15.78:0/3592222434 learned_addr learned my addr 172.21.15.78:0/3592222434 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 -- 172.21.15.78:0/3592222434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 msgr2=0x56329254f770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254f770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 -- 172.21.15.78:0/3592222434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 msgr2=0x5632925520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x5632925520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 -- 172.21.15.78:0/3592222434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563292450340 con 0x563292544af0 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f5343fff640 1 -- 172.21.15.78:0/3323144454 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f534401c070 con 0x7f534c100a90 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f534b59e640 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f533c004a00 tx=0x7f533c004a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b95fd7fba19045 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.026+0000 7f53417fa640 1 -- 172.21.15.78:0/3592222434 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c04f070 con 0x563292544af0 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f53417fa640 1 -- 172.21.15.78:0/3592222434 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c005430 con 0x563292544af0 2024-09-16T01:12:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f53417fa640 1 -- 172.21.15.78:0/3592222434 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c046030 con 0x563292544af0 2024-09-16T01:12:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f5343fff640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 0x7f532c0a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:22 smithi078 ceph-mon[30937]: pgmap v50: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 12 MiB/s wr, 845 op/s 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f534ad9d640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 0x7f532c0a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f5343fff640 1 -- 172.21.15.78:0/3323144454 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f53440cf4e0 con 0x7f534c100a90 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f5353fbe740 1 -- 172.21.15.78:0/3592222434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 msgr2=0x563292544ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.027+0000 7f5353fbe740 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f533c004a00 tx=0x7f533c004a30 comp rx=0 tx=0).stop 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 -- 172.21.15.78:0/3592222434 shutdown_connections 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x5632925520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 --2- 172.21.15.78:0/3592222434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x563292544ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 -- 172.21.15.78:0/3592222434 >> 172.21.15.78:0/3592222434 conn(0x563292541060 msgr2=0x563292541440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 -- 172.21.15.78:0/3592222434 shutdown_connections 2024-09-16T01:12:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 -- 172.21.15.78:0/3592222434 wait complete. 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f534ad9d640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 0x7f532c0a59f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x563292541ec0 tx=0x7f53440077b0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 -- start start 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.028+0000 7f5353fbe740 1 Processor -- start 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x56329254cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x56329254d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563292552920 con 0x56329254fcb0 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563292432290 con 0x563292545410 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563292552aa0 con 0x563292544af0 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f5353fbe740 10 client.0 _ll_register_callbacks cb 0x563292463140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f534a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f534a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43788/0 (socket says 172.21.15.78:43788) 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f534a59c640 1 -- 172.21.15.78:0/2946305733 learned_addr learned my addr 172.21.15.78:0/2946305733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f534b59e640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x56329254cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.029+0000 7f534b59e640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x56329254d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 -- 172.21.15.78:0/2946305733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 msgr2=0x56329254cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x56329254cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 -- 172.21.15.78:0/2946305733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 msgr2=0x56329254d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x56329254d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 -- 172.21.15.78:0/2946305733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632921bc210 con 0x563292545410 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f534a59c640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f533400e650 tx=0x7f533400eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed36a0e78636b25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f533a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533400aa90 con 0x563292545410 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f533400ac30 con 0x563292545410 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f533c005430 con 0x563292545410 2024-09-16T01:12:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5334023050 con 0x563292545410 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f5353fbe740 10 client.25079 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563292288620 con 0x563292545410 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.030+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c005430 con 0x563292545410 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5334010430 con 0x563292545410 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 10 client.25079 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 10 client.25079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 20 client.25079 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56329255fd90 0x5632925621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 4 client.25079 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632924b6a20 con 0x56329255fd90 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f534b59e640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56329255fd90 0x5632925621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f533a7fc640 10 client.25079 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.031+0000 7f534b59e640 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56329255fd90 0x5632925621b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ada88e00f3664e9c server_cookie=3eebd2ea5d19bb20 in_seq=0 out_seq=0 2024-09-16T01:12:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.032+0000 7f533a7fc640 1 -- 172.21.15.78:0/2946305733 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632924b6a20 con 0x56329255fd90 2024-09-16T01:12:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.032+0000 7f533a7fc640 10 client.25079 handle_command_reply: tid=0 2024-09-16T01:12:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.032+0000 7f5353fbe740 1 client.25079 shutdown 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.032+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 msgr2=0x56329254cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.032+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f533400e650 tx=0x7f533400eb10 comp rx=0 tx=0).stop 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 shutdown_connections 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56329255fd90 0x5632925621b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x56329254d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x56329254cff0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2946305733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292544af0 0x56329254cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 >> 172.21.15.78:0/2946305733 conn(0x563292541060 msgr2=0x563292546d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.033+0000 7f533a7fc640 0 client.25079 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.034+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 shutdown_connections 2024-09-16T01:12:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.034+0000 7f5353fbe740 1 -- 172.21.15.78:0/2946305733 wait complete. 2024-09-16T01:12:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.034+0000 7f5353fbe740 20 client.25079 trim_cache size 0 max 16384 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 Processor -- start 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 -- start start 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x563292567ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56329254fcb0 0x563292568000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f53300127e0 con 0x563292544af0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5330010080 con 0x563292545410 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f5353fbe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f533000ded0 con 0x56329254fcb0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f534b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f534b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48244/0 (socket says 172.21.15.78:48244) 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f534a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x563292567ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f534b59e640 1 -- 172.21.15.78:0/1291227543 learned_addr learned my addr 172.21.15.78:0/1291227543 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.052+0000 7f534b59e640 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56329254fcb0 0x563292568000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 -- 172.21.15.78:0/1291227543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56329254fcb0 msgr2=0x563292568000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56329254fcb0 0x563292568000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 -- 172.21.15.78:0/1291227543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 msgr2=0x563292567ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x563292567ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 -- 172.21.15.78:0/1291227543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563292540e90 con 0x563292544af0 2024-09-16T01:12:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534a59c640 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x563292567ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f534b59e640 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f533c006010 tx=0x7f533c0405b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac361b0f7788fbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f5340ff9640 1 -- 172.21.15.78:0/1291227543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c046040 con 0x563292544af0 2024-09-16T01:12:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f5340ff9640 1 -- 172.21.15.78:0/1291227543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c040eb0 con 0x563292544af0 2024-09-16T01:12:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.053+0000 7f5340ff9640 1 -- 172.21.15.78:0/1291227543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c04f040 con 0x563292544af0 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- 172.21.15.78:0/1291227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 msgr2=0x56329254d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f533c006010 tx=0x7f533c0405b0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- 172.21.15.78:0/1291227543 shutdown_connections 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56329254fcb0 0x563292568000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292545410 0x563292567ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- 172.21.15.78:0/1291227543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563292544af0 0x56329254d360 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- 172.21.15.78:0/1291227543 >> 172.21.15.78:0/1291227543 conn(0x563292541060 msgr2=0x56329254bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- 172.21.15.78:0/1291227543 shutdown_connections 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- 172.21.15.78:0/1291227543 wait complete. 2024-09-16T01:12:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- start start 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 Processor -- start 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292544af0 0x563292558340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292545410 0x5632925508f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f533000e540 con 0x56329254fcb0 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f533000f9b0 con 0x563292544af0 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f533000ec10 con 0x563292545410 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f5353fbe740 10 client.0 _ll_register_callbacks cb 0x7f533400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.054+0000 7f534b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48250/0 (socket says 172.21.15.78:48250) 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 -- 172.21.15.78:0/2886126934 learned_addr learned my addr 172.21.15.78:0/2886126934 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534a59c640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292545410 0x5632925508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292544af0 0x563292558340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 -- 172.21.15.78:0/2886126934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292545410 msgr2=0x5632925508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292545410 0x5632925508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 -- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292544af0 msgr2=0x563292558340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292544af0 0x563292558340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 -- 172.21.15.78:0/2886126934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563292551420 con 0x56329254fcb0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f534b59e640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5632925501e0 tx=0x7f533c005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff329d64ab38ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f533a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c046030 con 0x56329254fcb0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f533c05e070 con 0x56329254fcb0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5632924b0d70 con 0x56329254fcb0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f533c068030 con 0x56329254fcb0 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f5353fbe740 10 client.15600 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.055+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563292555bb0 con 0x56329254fcb0 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.056+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5632924b0d70 con 0x56329254fcb0 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f533c054930 con 0x56329254fcb0 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 10 client.15600 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 10 client.15600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 20 client.15600 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5632925cab90 0x5632925ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 4 client.15600 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56329242d900 con 0x5632925cab90 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f534a59c640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5632925cab90 0x5632925ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.057+0000 7f533a7fc640 10 client.15600 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.058+0000 7f534a59c640 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5632925cab90 0x5632925ccf80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2003178a55f11006 server_cookie=c0b9ecc0d9e84752 in_seq=0 out_seq=0 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f533a7fc640 1 -- 172.21.15.78:0/2886126934 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56329242d900 con 0x5632925cab90 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f533a7fc640 10 client.15600 handle_command_reply: tid=0 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 client.15600 shutdown 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 msgr2=0x563292550e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5632925501e0 tx=0x7f533c005320 comp rx=0 tx=0).stop 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 shutdown_connections 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5632925cab90 0x5632925ccf80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f533a7fc640 0 client.15600 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56329254fcb0 0x563292550e30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563292545410 0x5632925508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 --2- 172.21.15.78:0/2886126934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563292544af0 0x563292558340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 >> 172.21.15.78:0/2886126934 conn(0x563292541060 msgr2=0x563292546550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 shutdown_connections 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 1 -- 172.21.15.78:0/2886126934 wait complete. 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.065+0000 7f5353fbe740 20 client.15600 trim_cache size 0 max 16384 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:12:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:12:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 msgr2=0x7f532c0a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 0x7f532c0a59f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x563292541ec0 tx=0x7f53440077b0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 msgr2=0x7f534c114cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f534400ba80 tx=0x7f534400bf40 comp rx=0 tx=0).stop 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 shutdown_connections 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f532c0a3530 0x7f532c0a59f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f534c105f10 0x7f534c111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f534c1013b0 0x7f534c110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 --2- 172.21.15.78:0/3323144454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f534c100a90 0x7f534c114cc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 >> 172.21.15.78:0/3323144454 conn(0x7f534c0fc6b0 msgr2=0x7f534c103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 shutdown_connections 2024-09-16T01:12:23.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:23.066+0000 7f5351223640 1 -- 172.21.15.78:0/3323144454 wait complete. 2024-09-16T01:12:24.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:24 smithi193 ceph-mon[35701]: pgmap v51: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 MiB/s wr, 1.16k op/s 2024-09-16T01:12:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:24 smithi148 ceph-mon[35030]: pgmap v51: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 MiB/s wr, 1.16k op/s 2024-09-16T01:12:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:24 smithi078 ceph-mon[30937]: pgmap v51: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 MiB/s wr, 1.16k op/s 2024-09-16T01:12:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 70.0 s, 3388.0 tps, lat 29.307 ms stddev 31.467 2024-09-16T01:12:26.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:26 smithi193 ceph-mon[35701]: pgmap v52: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:26.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:26 smithi148 ceph-mon[35030]: pgmap v52: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:26 smithi078 ceph-mon[30937]: pgmap v52: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:28.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:28 smithi148 ceph-mon[35030]: pgmap v53: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:29.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:28 smithi078 ceph-mon[30937]: pgmap v53: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:29.045 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:28 smithi193 ceph-mon[35701]: pgmap v53: 129 pgs: 129 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-09-16T01:12:30.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:30 smithi148 ceph-mon[35030]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-09-16T01:12:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 75.0 s, 2679.6 tps, lat 37.526 ms stddev 46.752 2024-09-16T01:12:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:30 smithi078 ceph-mon[30937]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-09-16T01:12:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:30 smithi193 ceph-mon[35701]: pgmap v54: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 16 MiB/s wr, 1.12k op/s 2024-09-16T01:12:32.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:32 smithi148 ceph-mon[35030]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 601 op/s 2024-09-16T01:12:33.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:32 smithi078 ceph-mon[30937]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 601 op/s 2024-09-16T01:12:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:32 smithi193 ceph-mon[35701]: pgmap v55: 129 pgs: 129 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 601 op/s 2024-09-16T01:12:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:34 smithi148 ceph-mon[35030]: pgmap v56: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:34 smithi078 ceph-mon[30937]: pgmap v56: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:34 smithi193 ceph-mon[35701]: pgmap v56: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 80.0 s, 2832.6 tps, lat 35.230 ms stddev 36.594 2024-09-16T01:12:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.190+0000 7fb524903640 1 -- 172.21.15.78:0/2255333505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 msgr2=0x7fb520101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 --2- 172.21.15.78:0/2255333505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb520101f30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb508009930 tx=0x7fb50802a420 comp rx=0 tx=0).stop 2024-09-16T01:12:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 -- 172.21.15.78:0/2255333505 shutdown_connections 2024-09-16T01:12:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 --2- 172.21.15.78:0/2255333505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 --2- 172.21.15.78:0/2255333505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb520101f30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 --2- 172.21.15.78:0/2255333505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 0x7fb520107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 -- 172.21.15.78:0/2255333505 >> 172.21.15.78:0/2255333505 conn(0x7fb5200fd7b0 msgr2=0x7fb5200ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 -- 172.21.15.78:0/2255333505 shutdown_connections 2024-09-16T01:12:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.191+0000 7fb524903640 1 -- 172.21.15.78:0/2255333505 wait complete. 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.192+0000 7fb524903640 1 Processor -- start 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 -- start start 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 0x7fb52019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb520111ee0 con 0x7fb520101ad0 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb520111d60 con 0x7fb520107b70 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb524903640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb520112060 con 0x7fb520102470 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51084/0 (socket says 172.21.15.78:51084) 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb51e59c640 1 -- 172.21.15.78:0/3970878759 learned_addr learned my addr 172.21.15.78:0/3970878759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.193+0000 7fb517fff640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 0x7fb52019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51e59c640 1 -- 172.21.15.78:0/3970878759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 msgr2=0x7fb52019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51e59c640 1 -- 172.21.15.78:0/3970878759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 msgr2=0x7fb52019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 0x7fb52019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb51e59c640 1 -- 172.21.15.78:0/3970878759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52019f5d0 con 0x7fb520101ad0 2024-09-16T01:12:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.194+0000 7fb517fff640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.195+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb510009f50 tx=0x7fb51000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.195+0000 7fb5177fe640 1 -- 172.21.15.78:0/3970878759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510024070 con 0x7fb520101ad0 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.195+0000 7fb5177fe640 1 -- 172.21.15.78:0/3970878759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51001f440 con 0x7fb520101ad0 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.195+0000 7fb5177fe640 1 -- 172.21.15.78:0/3970878759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51001d5a0 con 0x7fb520101ad0 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.195+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52019f8c0 con 0x7fb520101ad0 2024-09-16T01:12:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.196+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb520108930 con 0x7fb520101ad0 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.198+0000 7fb5177fe640 1 -- 172.21.15.78:0/3970878759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb510027070 con 0x7fb520101ad0 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.198+0000 7fb52769b740 1 Processor -- start 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.198+0000 7fb52769b740 1 -- start start 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.198+0000 7fb52769b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c16871bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.198+0000 7fb52769b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb52769b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c168726cb0 0x55c1687290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb52769b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c1686802f0 con 0x55c16871c410 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb52769b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c168628270 con 0x55c168726cb0 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb52769b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c16867c2b0 con 0x55c16871baf0 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51092/0 (socket says 172.21.15.78:51092) 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 -- 172.21.15.78:0/2003174546 learned_addr learned my addr 172.21.15.78:0/2003174546 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb517fff640 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c168726cb0 0x55c1687290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c16871bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 -- 172.21.15.78:0/2003174546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 msgr2=0x55c16871bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c16871bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 -- 172.21.15.78:0/2003174546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c168726cb0 msgr2=0x55c1687290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c168726cb0 0x55c1687290a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.199+0000 7fb51e59c640 1 -- 172.21.15.78:0/2003174546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c168627340 con 0x55c16871c410 2024-09-16T01:12:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb5177fe640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 0x7fb50c0a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb5100d5750 tx=0x7fb5100e0410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178708fbe6a1e6a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb5177fe640 1 -- 172.21.15.78:0/3970878759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb5100d83a0 con 0x7fb520101ad0 2024-09-16T01:12:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb514ff9640 1 -- 172.21.15.78:0/2003174546 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a06f0 con 0x55c16871c410 2024-09-16T01:12:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb514ff9640 1 -- 172.21.15.78:0/2003174546 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5100a4450 con 0x55c16871c410 2024-09-16T01:12:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb514ff9640 1 -- 172.21.15.78:0/2003174546 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a3db0 con 0x55c16871c410 2024-09-16T01:12:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.201+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 0x7fb50c0a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 -- 172.21.15.78:0/2003174546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 msgr2=0x55c168726770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb5100d5750 tx=0x7fb5100e0410 comp rx=0 tx=0).stop 2024-09-16T01:12:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 0x7fb50c0a5b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb518007ed0 tx=0x7fb518007680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 -- 172.21.15.78:0/2003174546 shutdown_connections 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c168726cb0 0x55c1687290a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c168726770 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 --2- 172.21.15.78:0/2003174546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c16871bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 -- 172.21.15.78:0/2003174546 >> 172.21.15.78:0/2003174546 conn(0x55c168718060 msgr2=0x55c168718440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 -- 172.21.15.78:0/2003174546 shutdown_connections 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.202+0000 7fb52769b740 1 -- 172.21.15.78:0/2003174546 wait complete. 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 -- start start 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 Processor -- start 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c168724160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c1687246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c168729880 con 0x55c16871baf0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c168609290 con 0x55c16871c410 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c168729a00 con 0x55c168726cb0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb52769b740 10 client.0 _ll_register_callbacks cb 0x55c16863a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51096/0 (socket says 172.21.15.78:51096) 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb51e59c640 1 -- 172.21.15.78:0/3671311157 learned_addr learned my addr 172.21.15.78:0/3671311157 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c1687246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.203+0000 7fb517fff640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c168724160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 -- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 msgr2=0x55c168724160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c168724160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 -- 172.21.15.78:0/3671311157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 msgr2=0x55c168723c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 -- 172.21.15.78:0/3671311157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c168393210 con 0x55c168726cb0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb517fff640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c168724160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c1687246a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb518011ca0 tx=0x7fb518004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a966bcb38a452157 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb505ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51801cd30 con 0x55c168726cb0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5100a4450 con 0x55c168726cb0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.204+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb51801d410 con 0x55c168726cb0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb51801d610 con 0x55c168726cb0 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 10 client.25174 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c16845f620 con 0x55c168726cb0 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51801dc90 con 0x55c168726cb0 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fb51801a990 con 0x55c168726cb0 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 10 client.25174 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 10 client.25174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 20 client.25174 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.205+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687370c0 0x55c168739500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.206+0000 7fb52769b740 4 client.25174 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:12:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.206+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb5100a4450 con 0x55c1687370c0 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.206+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687370c0 0x55c168739500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.207+0000 7fb505ffb640 10 client.25174 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.207+0000 7fb51e59c640 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687370c0 0x55c168739500 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7837d6c2d30ba0ed server_cookie=eb7ce5087b5151a5 in_seq=0 out_seq=0 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb505ffb640 1 -- 172.21.15.78:0/3671311157 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb5100a4450 con 0x55c1687370c0 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb505ffb640 10 client.25174 handle_command_reply: tid=0 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 client.25174 shutdown 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 msgr2=0x55c1687246a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c1687246a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb518011ca0 tx=0x7fb518004a40 comp rx=0 tx=0).stop 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 shutdown_connections 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687370c0 0x55c168739500 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c1687246a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c168724160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 --2- 172.21.15.78:0/3671311157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871baf0 0x55c168723c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 >> 172.21.15.78:0/3671311157 conn(0x55c168718060 msgr2=0x55c1687171c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb505ffb640 0 client.25174 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.208+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 shutdown_connections 2024-09-16T01:12:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.209+0000 7fb52769b740 1 -- 172.21.15.78:0/3671311157 wait complete. 2024-09-16T01:12:36.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.209+0000 7fb52769b740 20 client.25174 trim_cache size 0 max 16384 2024-09-16T01:12:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 Processor -- start 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 -- start start 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c1687244d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c16873ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 0x55c16873f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201af6c0 con 0x55c168726cb0 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201acf60 con 0x55c16871c410 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb52769b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201aadd0 con 0x55c16871baf0 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb51ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 0x55c16873f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb517fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c16873ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb51ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 0x55c16873f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51110/0 (socket says 172.21.15.78:51110) 2024-09-16T01:12:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb51ed9d640 1 -- 172.21.15.78:0/2902424393 learned_addr learned my addr 172.21.15.78:0/2902424393 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.235+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c1687244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 -- 172.21.15.78:0/2902424393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 msgr2=0x55c16873ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c16873ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 -- 172.21.15.78:0/2902424393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 msgr2=0x55c16873f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 0x55c16873f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 -- 172.21.15.78:0/2902424393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c168733660 con 0x55c16871baf0 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c1687244d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5100acae0 tx=0x7fb5100c4040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3577a05eaba5b29b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb507fff640 1 -- 172.21.15.78:0/2902424393 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a3f90 con 0x55c16871baf0 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb507fff640 1 -- 172.21.15.78:0/2902424393 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5100d5e50 con 0x55c16871baf0 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.236+0000 7fb507fff640 1 -- 172.21.15.78:0/2902424393 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a4220 con 0x55c16871baf0 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 -- 172.21.15.78:0/2902424393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 msgr2=0x55c1687244d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c1687244d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5100acae0 tx=0x7fb5100c4040 comp rx=0 tx=0).stop 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 -- 172.21.15.78:0/2902424393 shutdown_connections 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c168726cb0 0x55c16873f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871c410 0x55c16873ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 --2- 172.21.15.78:0/2902424393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16871baf0 0x55c1687244d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 -- 172.21.15.78:0/2902424393 >> 172.21.15.78:0/2902424393 conn(0x55c168718060 msgr2=0x55c168718440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.237+0000 7fb52769b740 1 -- 172.21.15.78:0/2902424393 shutdown_connections 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.238+0000 7fb52769b740 1 -- 172.21.15.78:0/2902424393 wait complete. 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.238+0000 7fb52769b740 1 -- start start 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.238+0000 7fb52769b740 1 Processor -- start 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871baf0 0x55c16872ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c16872f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c168720430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201ab420 con 0x55c16871c410 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201ac890 con 0x55c16871baf0 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5201abaf0 con 0x55c168726cb0 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb52769b740 10 client.0 _ll_register_callbacks cb 0x7fb518007920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871baf0 0x55c16872ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c168720430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c16872f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c16872f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51124/0 (socket says 172.21.15.78:51124) 2024-09-16T01:12:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb517fff640 1 -- 172.21.15.78:0/2760394623 learned_addr learned my addr 172.21.15.78:0/2760394623 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 -- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871baf0 msgr2=0x55c16872ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871baf0 0x55c16872ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 -- 172.21.15.78:0/2760394623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 msgr2=0x55c16872f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c16872f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.239+0000 7fb51ed9d640 1 -- 172.21.15.78:0/2760394623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c168720b70 con 0x55c168726cb0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb51ed9d640 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c168720430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb518007b90 tx=0x7fb518004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cca72e60b52a052 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb505ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51801d9c0 con 0x55c168726cb0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb51801db60 con 0x55c168726cb0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5100d5e50 con 0x55c168726cb0 2024-09-16T01:12:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb51801dd60 con 0x55c168726cb0 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb52769b740 10 client.25180 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.240+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1687a17f0 con 0x55c168726cb0 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5100d5e50 con 0x55c168726cb0 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fb51801a870 con 0x55c168726cb0 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 10 client.25180 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 10 client.25180 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 20 client.25180 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687a1eb0 0x55c1687a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 4 client.25180 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:12:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.241+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c16868da20 con 0x55c1687a1eb0 2024-09-16T01:12:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.242+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687a1eb0 0x55c1687a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.243+0000 7fb51e59c640 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687a1eb0 0x55c1687a42d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b808b2d3a62eeb4 server_cookie=2a4d2436b3d509f1 in_seq=0 out_seq=0 2024-09-16T01:12:36.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.243+0000 7fb505ffb640 10 client.25180 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb505ffb640 1 -- 172.21.15.78:0/2760394623 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c16868da20 con 0x55c1687a1eb0 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb505ffb640 10 client.25180 handle_command_reply: tid=0 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 client.25180 shutdown 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 msgr2=0x55c168720430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c168720430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb518007b90 tx=0x7fb518004ce0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 shutdown_connections 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c1687a1eb0 0x55c1687a42d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c168726cb0 0x55c168720430 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c16871c410 0x55c16872f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 --2- 172.21.15.78:0/2760394623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c16871baf0 0x55c16872ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 >> 172.21.15.78:0/2760394623 conn(0x55c168718060 msgr2=0x55c168725480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb505ffb640 0 client.25180 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 shutdown_connections 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 1 -- 172.21.15.78:0/2760394623 wait complete. 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.249+0000 7fb52769b740 20 client.25180 trim_cache size 0 max 16384 2024-09-16T01:12:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:12:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:12:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:12:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:12:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 msgr2=0x7fb50c0a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 0x7fb50c0a5b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb518007ed0 tx=0x7fb518007680 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 msgr2=0x7fb52019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb510009f50 tx=0x7fb51000ee50 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 shutdown_connections 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb50c0a3640 0x7fb50c0a5b00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb520107b70 0x7fb52019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb520102470 0x7fb52019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 --2- 172.21.15.78:0/3970878759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb520101ad0 0x7fb52019a5e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:36.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 >> 172.21.15.78:0/3970878759 conn(0x7fb5200fd7b0 msgr2=0x7fb52010d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 shutdown_connections 2024-09-16T01:12:36.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:36.250+0000 7fb524903640 1 -- 172.21.15.78:0/3970878759 wait complete. 2024-09-16T01:12:36.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:36 smithi078 ceph-mon[30937]: pgmap v57: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:36.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:36 smithi148 ceph-mon[35030]: pgmap v57: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:36 smithi193 ceph-mon[35701]: pgmap v57: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:38.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:38 smithi148 ceph-mon[35030]: pgmap v58: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:38 smithi078 ceph-mon[30937]: pgmap v58: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:38 smithi193 ceph-mon[35701]: pgmap v58: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 719 op/s 2024-09-16T01:12:40.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:40 smithi148 ceph-mon[35030]: pgmap v59: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 85.0 s, 3125.4 tps, lat 31.979 ms stddev 36.384 2024-09-16T01:12:41.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:40 smithi078 ceph-mon[30937]: pgmap v59: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:40 smithi193 ceph-mon[35701]: pgmap v59: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 MiB/s wr, 1.03k op/s 2024-09-16T01:12:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:42 smithi148 ceph-mon[35030]: pgmap v60: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 739 op/s 2024-09-16T01:12:43.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:42 smithi078 ceph-mon[30937]: pgmap v60: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 739 op/s 2024-09-16T01:12:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:42 smithi193 ceph-mon[35701]: pgmap v60: 129 pgs: 129 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 739 op/s 2024-09-16T01:12:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:44 smithi148 ceph-mon[35030]: pgmap v61: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.01k op/s 2024-09-16T01:12:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:44 smithi078 ceph-mon[30937]: pgmap v61: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.01k op/s 2024-09-16T01:12:45.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:44 smithi193 ceph-mon[35701]: pgmap v61: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.01k op/s 2024-09-16T01:12:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 90.0 s, 2953.0 tps, lat 33.816 ms stddev 37.078 2024-09-16T01:12:46.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:46 smithi148 ceph-mon[35030]: pgmap v62: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:47.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:46 smithi078 ceph-mon[30937]: pgmap v62: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:47.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:46 smithi193 ceph-mon[35701]: pgmap v62: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:48.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:48 smithi148 ceph-mon[35030]: pgmap v63: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:49.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:48 smithi078 ceph-mon[30937]: pgmap v63: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:49.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:48 smithi193 ceph-mon[35701]: pgmap v63: 129 pgs: 129 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.1 MiB/s wr, 598 op/s 2024-09-16T01:12:49.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/1663951443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04101ad0 msgr2=0x7f5c04101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/1663951443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04101ad0 0x7f5c04101f30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009860 tx=0x7f5bf402a340 comp rx=0 tx=0).stop 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/1663951443 shutdown_connections 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/1663951443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04102470 0x7f5c0410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/1663951443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04101ad0 0x7f5c04101f30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/1663951443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04107b70 0x7f5c04107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.373+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/1663951443 >> 172.21.15.78:0/1663951443 conn(0x7f5c040fd7d0 msgr2=0x7f5c040ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/1663951443 shutdown_connections 2024-09-16T01:12:49.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/1663951443 wait complete. 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 Processor -- start 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- start start 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04102470 0x7f5c04198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 0x7f5c0419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04111ee0 con 0x7f5c04101ad0 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04111d60 con 0x7f5c04107b70 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.374+0000 7f5c0bbe8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04112060 con 0x7f5c04102470 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53322/0 (socket says 172.21.15.78:53322) 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c0a185640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 0x7f5c0419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 -- 172.21.15.78:0/879540779 learned_addr learned my addr 172.21.15.78:0/879540779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09183640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04102470 0x7f5c04198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 -- 172.21.15.78:0/879540779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04102470 msgr2=0x7f5c04198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04102470 0x7f5c04198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 -- 172.21.15.78:0/879540779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 msgr2=0x7f5c0419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 0x7f5c0419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 -- 172.21.15.78:0/879540779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0419d340 con 0x7f5c04101ad0 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c0a185640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 0x7f5c0419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c09984640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000e9f0 tx=0x7f5c0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5bfaffd640 1 -- 172.21.15.78:0/879540779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000cd80 con 0x7f5c04101ad0 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5bfaffd640 1 -- 172.21.15.78:0/879540779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c00010910 con 0x7f5c04101ad0 2024-09-16T01:12:49.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0419d630 con 0x7f5c04101ad0 2024-09-16T01:12:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5bfaffd640 1 -- 172.21.15.78:0/879540779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00018990 con 0x7f5c04101ad0 2024-09-16T01:12:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.375+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0419da40 con 0x7f5c04101ad0 2024-09-16T01:12:49.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.387+0000 7f5c0ea61740 1 Processor -- start 2024-09-16T01:12:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 -- start start 2024-09-16T01:12:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5593a75032f0 con 0x5593a759f410 2024-09-16T01:12:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.388+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5593a74ab270 con 0x5593a75a9cb0 2024-09-16T01:12:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.389+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593a74ff2b0 con 0x5593a759eaf0 2024-09-16T01:12:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.389+0000 7f5bfaffd640 1 -- 172.21.15.78:0/879540779 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f5c00018b30 con 0x7f5c04101ad0 2024-09-16T01:12:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.390+0000 7f5bfaffd640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 0x7f5bf00a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.393+0000 7f5c09984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.393+0000 7f5c09984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53326/0 (socket says 172.21.15.78:53326) 2024-09-16T01:12:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.393+0000 7f5c09984640 1 -- 172.21.15.78:0/1854744105 learned_addr learned my addr 172.21.15.78:0/1854744105 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.393+0000 7f5bfaffd640 1 -- 172.21.15.78:0/879540779 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5c000bc060 con 0x7f5c04101ad0 2024-09-16T01:12:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.394+0000 7f5c09984640 1 -- 172.21.15.78:0/1854744105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 msgr2=0x5593a759eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.394+0000 7f5c09984640 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.394+0000 7f5c09984640 1 -- 172.21.15.78:0/1854744105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 msgr2=0x5593a75ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.395+0000 7f5c09984640 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.395+0000 7f5c09984640 1 -- 172.21.15.78:0/1854744105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a74aa340 con 0x5593a759f410 2024-09-16T01:12:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.395+0000 7f5c09984640 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5c00099290 tx=0x7f5c000a14e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f626fad119b899 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.396+0000 7f5c0a185640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 0x7f5bf00a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.396+0000 7f5beffff640 1 -- 172.21.15.78:0/1854744105 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00098a40 con 0x5593a759f410 2024-09-16T01:12:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.396+0000 7f5beffff640 1 -- 172.21.15.78:0/1854744105 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c0009c710 con 0x5593a759f410 2024-09-16T01:12:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.396+0000 7f5beffff640 1 -- 172.21.15.78:0/1854744105 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00098be0 con 0x5593a759f410 2024-09-16T01:12:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.399+0000 7f5c0ea61740 1 -- 172.21.15.78:0/1854744105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 msgr2=0x5593a75a9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.399+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5c00099290 tx=0x7f5c000a14e0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.401+0000 7f5c0a185640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 0x7f5bf00a5920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005fd0 tx=0x7f5bfc010040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.406+0000 7f5c0ea61740 1 -- 172.21.15.78:0/1854744105 shutdown_connections 2024-09-16T01:12:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.406+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.406+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a9770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.406+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/1854744105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.406+0000 7f5c0ea61740 1 -- 172.21.15.78:0/1854744105 >> 172.21.15.78:0/1854744105 conn(0x5593a759b060 msgr2=0x5593a759b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.407+0000 7f5c0ea61740 1 -- 172.21.15.78:0/1854744105 shutdown_connections 2024-09-16T01:12:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.411+0000 7f5c0ea61740 1 -- 172.21.15.78:0/1854744105 wait complete. 2024-09-16T01:12:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.411+0000 7f5c0ea61740 1 -- start start 2024-09-16T01:12:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.411+0000 7f5c0ea61740 1 Processor -- start 2024-09-16T01:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5593a75bd800 con 0x5593a759f410 2024-09-16T01:12:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5593a748c290 con 0x5593a75a9cb0 2024-09-16T01:12:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593a75bd980 con 0x5593a759eaf0 2024-09-16T01:12:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.413+0000 7f5c0ea61740 10 client.0 _ll_register_callbacks cb 0x5593a74bd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.415+0000 7f5c0a185640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.417+0000 7f5c0a185640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41066/0 (socket says 172.21.15.78:41066) 2024-09-16T01:12:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.417+0000 7f5c0a185640 1 -- 172.21.15.78:0/3776621195 learned_addr learned my addr 172.21.15.78:0/3776621195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.418+0000 7f5c09984640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.418+0000 7f5c09183640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 -- 172.21.15.78:0/3776621195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 msgr2=0x5593a75a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 -- 172.21.15.78:0/3776621195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 msgr2=0x5593a75a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 -- 172.21.15.78:0/3776621195 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a7216210 con 0x5593a75a9cb0 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.419+0000 7f5c0a185640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00cb40 tx=0x7f5bfc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ced94d7afd8503c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.420+0000 7f5bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:12:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.420+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc004810 con 0x5593a75a9cb0 2024-09-16T01:12:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.420+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5bfc024400 con 0x5593a75a9cb0 2024-09-16T01:12:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.421+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c0009c710 con 0x5593a75a9cb0 2024-09-16T01:12:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.430+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bfc024600 con 0x5593a75a9cb0 2024-09-16T01:12:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.430+0000 7f5c0ea61740 10 client.25112 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.430+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593a72e2620 con 0x5593a75a9cb0 2024-09-16T01:12:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c0009c710 con 0x5593a75a9cb0 2024-09-16T01:12:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5bfc00b040 con 0x5593a75a9cb0 2024-09-16T01:12:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 10 client.25112 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 10 client.25112 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 20 client.25112 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a75cad00 0x5593a75cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 4 client.25112 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:12:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.433+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593a7510a20 con 0x5593a75cad00 2024-09-16T01:12:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.436+0000 7f5c09984640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a75cad00 0x5593a75cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.437+0000 7f5c09984640 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a75cad00 0x5593a75cd110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0833342c94c436 server_cookie=435c0b9c128d7ac9 in_seq=0 out_seq=0 2024-09-16T01:12:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.438+0000 7f5bed7fa640 10 client.25112 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.439+0000 7f5bed7fa640 1 -- 172.21.15.78:0/3776621195 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593a7510a20 con 0x5593a75cad00 2024-09-16T01:12:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.439+0000 7f5bed7fa640 10 client.25112 handle_command_reply: tid=0 2024-09-16T01:12:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.439+0000 7f5c0ea61740 1 client.25112 shutdown 2024-09-16T01:12:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.441+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 msgr2=0x5593a75a76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.441+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00cb40 tx=0x7f5bfc004a40 comp rx=0 tx=0).stop 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 shutdown_connections 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a75cad00 0x5593a75cd110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75a76d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a75a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3776621195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 >> 172.21.15.78:0/3776621195 conn(0x5593a759b060 msgr2=0x5593a759a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.444+0000 7f5bed7fa640 0 client.25112 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.447+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 shutdown_connections 2024-09-16T01:12:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.449+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3776621195 wait complete. 2024-09-16T01:12:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.449+0000 7f5c0ea61740 20 client.25112 trim_cache size 0 max 16384 2024-09-16T01:12:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 Processor -- start 2024-09-16T01:12:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 -- start start 2024-09-16T01:12:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75d2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5be8012900 con 0x5593a759f410 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5be80101a0 con 0x5593a75a9cb0 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.474+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5be800dff0 con 0x5593a759eaf0 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53340/0 (socket says 172.21.15.78:53340) 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42612/0 (socket says 172.21.15.78:42612) 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 -- 172.21.15.78:0/3317664384 learned_addr learned my addr 172.21.15.78:0/3317664384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c0a185640 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75d2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 -- 172.21.15.78:0/3317664384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 msgr2=0x5593a759ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 -- 172.21.15.78:0/3317664384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 msgr2=0x5593a75d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75d2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 -- 172.21.15.78:0/3317664384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a75a5be0 con 0x5593a759f410 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09984640 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.475+0000 7f5c09183640 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005d90 tx=0x7f5bf4009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f782c9957944fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.476+0000 7f5bef7fe640 1 -- 172.21.15.78:0/3317664384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4037040 con 0x5593a759f410 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.476+0000 7f5bef7fe640 1 -- 172.21.15.78:0/3317664384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf4030e40 con 0x5593a759f410 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.476+0000 7f5bef7fe640 1 -- 172.21.15.78:0/3317664384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4003970 con 0x5593a759f410 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3317664384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 msgr2=0x5593a759a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005d90 tx=0x7f5bf4009520 comp rx=0 tx=0).stop 2024-09-16T01:12:49.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3317664384 shutdown_connections 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a75a9cb0 0x5593a75d2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a759f410 0x5593a759a0b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/3317664384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a759ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3317664384 >> 172.21.15.78:0/3317664384 conn(0x5593a759b060 msgr2=0x5593a75999a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3317664384 shutdown_connections 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- 172.21.15.78:0/3317664384 wait complete. 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 -- start start 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.477+0000 7f5c0ea61740 1 Processor -- start 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 0x5593a75a40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5be800e660 con 0x5593a75a9cb0 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5be800fad0 con 0x5593a759f410 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5be800ed30 con 0x5593a759eaf0 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0a185640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c09984640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0a185640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53354/0 (socket says 172.21.15.78:53354) 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c09183640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 0x5593a75a40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0a185640 1 -- 172.21.15.78:0/2294077531 learned_addr learned my addr 172.21.15.78:0/2294077531 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.478+0000 7f5c0ea61740 10 client.0 _ll_register_callbacks cb 0x7f5bfc009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 -- 172.21.15.78:0/2294077531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 msgr2=0x5593a75a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 -- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 msgr2=0x5593a75a40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 0x5593a75a40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 -- 172.21.15.78:0/2294077531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a7634a40 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5c0a185640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc019cf0 tx=0x7f5bfc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65dab55f5c09bb2e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.479+0000 7f5bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5c09183640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 0x5593a75a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc03b060 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5bfc021040 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf4030e40 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bfc019400 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5c0ea61740 10 client.15636 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593a7635cf0 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.480+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf4030e40 con 0x5593a75a9cb0 2024-09-16T01:12:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5bfc02e420 con 0x5593a75a9cb0 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 10 client.15636 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 20 client.15636 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a7636450 0x5593a7638870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 4 client.15636 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593a750ad70 con 0x5593a7636450 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.481+0000 7f5c09984640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a7636450 0x5593a7638870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.482+0000 7f5c09984640 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a7636450 0x5593a7638870 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=495aa5b39700123a server_cookie=b1abb22e37d1cebb in_seq=0 out_seq=0 2024-09-16T01:12:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.482+0000 7f5bed7fa640 10 client.15636 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5bed7fa640 1 -- 172.21.15.78:0/2294077531 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593a750ad70 con 0x5593a7636450 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5bed7fa640 10 client.15636 handle_command_reply: tid=0 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 client.15636 shutdown 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 msgr2=0x5593a75a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc019cf0 tx=0x7f5bfc0095c0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 shutdown_connections 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5593a7636450 0x5593a7638870 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5bed7fa640 0 client.15636 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593a75a9cb0 0x5593a75a45f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5593a759f410 0x5593a75a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 --2- 172.21.15.78:0/2294077531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593a759eaf0 0x5593a75a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 >> 172.21.15.78:0/2294077531 conn(0x5593a759b060 msgr2=0x5593a75a7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 shutdown_connections 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.489+0000 7f5c0ea61740 1 -- 172.21.15.78:0/2294077531 wait complete. 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0ea61740 20 client.15636 trim_cache size 0 max 16384 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:12:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:12:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 msgr2=0x7f5bf00a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 0x7f5bf00a5920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc005fd0 tx=0x7f5bfc010040 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 msgr2=0x7f5c04198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000e9f0 tx=0x7f5c0000eeb0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 shutdown_connections 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f5bf00a3460 0x7f5bf00a5920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04107b70 0x7f5c0419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c04102470 0x7f5c04198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 --2- 172.21.15.78:0/879540779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c04101ad0 0x7f5c04198350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.490+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 >> 172.21.15.78:0/879540779 conn(0x7f5c040fd7d0 msgr2=0x7f5c040fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.491+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 shutdown_connections 2024-09-16T01:12:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:12:49.491+0000 7f5c0bbe8640 1 -- 172.21.15.78:0/879540779 wait complete. 2024-09-16T01:12:50.848 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:50 smithi193 ceph-mon[35701]: pgmap v64: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 889 op/s 2024-09-16T01:12:50.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:50 smithi148 ceph-mon[35030]: pgmap v64: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 889 op/s 2024-09-16T01:12:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 95.0 s, 2871.0 tps, lat 34.926 ms stddev 40.533 2024-09-16T01:12:51.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:50 smithi078 ceph-mon[30937]: pgmap v64: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.9 MiB/s wr, 889 op/s 2024-09-16T01:12:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: pgmap v65: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.9 MiB/s wr, 578 op/s 2024-09-16T01:12:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:12:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:52.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: pgmap v65: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.9 MiB/s wr, 578 op/s 2024-09-16T01:12:53.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:12:53.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: pgmap v65: 129 pgs: 129 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.9 MiB/s wr, 578 op/s 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:53.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:12:53.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:12:53.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:54.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:12:54.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:12:54.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:54.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:12:54.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:12:54.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:12:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:54 smithi148 ceph-mon[35030]: pgmap v66: 129 pgs: 129 active+clean; 1.2 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 813 op/s 2024-09-16T01:12:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:54 smithi078 ceph-mon[30937]: pgmap v66: 129 pgs: 129 active+clean; 1.2 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 813 op/s 2024-09-16T01:12:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:54 smithi193 ceph-mon[35701]: pgmap v66: 129 pgs: 129 active+clean; 1.2 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 813 op/s 2024-09-16T01:12:55.388 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T01:12:55.390 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-16T01:12:55.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:12:55.874 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:12:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 100.0 s, 2168.4 tps, lat 42.533 ms stddev 62.490 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.486+0000 7f0e01f96640 1 -- 172.21.15.78:0/1444907840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10e580 msgr2=0x7f0dfc10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.486+0000 7f0e01f96640 1 --2- 172.21.15.78:0/1444907840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10e580 0x7f0dfc10e960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009890 tx=0x7f0df002a3b0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 -- 172.21.15.78:0/1444907840 shutdown_connections 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 --2- 172.21.15.78:0/1444907840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 --2- 172.21.15.78:0/1444907840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dfc10ef30 0x7f0dfc10d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 --2- 172.21.15.78:0/1444907840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10e580 0x7f0dfc10e960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 -- 172.21.15.78:0/1444907840 >> 172.21.15.78:0/1444907840 conn(0x7f0dfc06e140 msgr2=0x7f0dfc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.487+0000 7f0e01f96640 1 -- 172.21.15.78:0/1444907840 shutdown_connections 2024-09-16T01:12:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.488+0000 7f0e01f96640 1 -- 172.21.15.78:0/1444907840 wait complete. 2024-09-16T01:12:56.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.488+0000 7f0e01f96640 1 Processor -- start 2024-09-16T01:12:56.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.489+0000 7f0e01f96640 1 -- start start 2024-09-16T01:12:56.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.489+0000 7f0e01f96640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:56.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.489+0000 7f0e01f96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dfc10e580 0x7f0dfc1a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:56.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0e01f96640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc19e850 0x7f0dfc19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0e01f96640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc118620 con 0x7f0dfc10e580 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0e01f96640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc1184a0 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0e01f96640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc1187a0 con 0x7f0dfc19e850 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59408/0 (socket says 172.21.15.78:59408) 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 -- 172.21.15.78:0/1220089466 learned_addr learned my addr 172.21.15.78:0/1220089466 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 -- 172.21.15.78:0/1220089466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc19e850 msgr2=0x7f0dfc19ed00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc19e850 0x7f0dfc19ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 -- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dfc10e580 msgr2=0x7f0dfc1a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dfc10e580 0x7f0dfc1a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.490+0000 7f0dfb7fe640 1 -- 172.21.15.78:0/1220089466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc19f430 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.491+0000 7f0dfb7fe640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006010 tx=0x7f0df0033d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.491+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0037040 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.492+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0df0030520 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.492+0000 7f0e01f96640 1 -- 172.21.15.78:0/1220089466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc19f660 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.492+0000 7f0e01f96640 1 -- 172.21.15.78:0/1220089466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1ab700 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.492+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df00055e0 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.494+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc073f10 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.498+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f0df0044020 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.498+0000 7f0df8ff9640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 0x7f0de40a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.499+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f0df00efe40 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.499+0000 7f0dfaffd640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 0x7f0de40a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.499+0000 7f0dfaffd640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 0x7f0de40a59c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc109500 tx=0x7f0df4005da0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.500+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0df003c3a0 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.845+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dfc062df0 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.847+0000 7f0df8ff9640 1 -- 172.21.15.78:0/1220089466 <== mon.2 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11922 (secure 0 0 0) 0x7f0df00bc540 con 0x7f0dfc10d9b0 2024-09-16T01:12:56.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:12:56.850 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":28,"btime":"2024-09-16T01:11:13:114118+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:11:12.780839+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:12:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 28 2024-09-16T01:12:56.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.851+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 msgr2=0x7f0de40a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:56.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.851+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 0x7f0de40a59c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc109500 tx=0x7f0df4005da0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.851+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 msgr2=0x7f0dfc1a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:56.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.851+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006010 tx=0x7f0df0033d80 comp rx=0 tx=0).stop 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 shutdown_connections 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f0de40a3500 0x7f0de40a59c0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc109500 tx=0x7f0df4005da0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc19e850 0x7f0dfc19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dfc10e580 0x7f0dfc1a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 --2- 172.21.15.78:0/1220089466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dfc10d9b0 0x7f0dfc1a46d0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006010 tx=0x7f0df0033d80 comp rx=0 tx=0).stop 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 >> 172.21.15.78:0/1220089466 conn(0x7f0dfc06e140 msgr2=0x7f0dfc072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 shutdown_connections 2024-09-16T01:12:56.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:56.852+0000 7f0de27fc640 1 -- 172.21.15.78:0/1220089466 wait complete. 2024-09-16T01:12:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:56 smithi148 ceph-mon[35030]: pgmap v67: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 596 op/s 2024-09-16T01:12:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:56 smithi078 ceph-mon[30937]: pgmap v67: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 596 op/s 2024-09-16T01:12:57.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:56 smithi193 ceph-mon[35701]: pgmap v67: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 596 op/s 2024-09-16T01:12:57.776 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-16T01:12:57.777 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-16T01:12:58.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:57 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1220089466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:12:58.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:57 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1220089466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:12:58.230 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:12:58.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:57 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1220089466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:12:59.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:12:58 smithi148 ceph-mon[35030]: pgmap v68: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.9 MiB/s wr, 582 op/s 2024-09-16T01:12:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:12:58 smithi078 ceph-mon[30937]: pgmap v68: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.9 MiB/s wr, 582 op/s 2024-09-16T01:12:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:12:58 smithi193 ceph-mon[35701]: pgmap v68: 129 pgs: 129 active+clean; 1.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.9 MiB/s wr, 582 op/s 2024-09-16T01:12:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.978+0000 7fcda64f0640 1 -- 172.21.15.78:0/2327089543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0104980 msgr2=0x7fcda01089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.978+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2327089543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0104980 0x7fcda01089d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009900 tx=0x7fcd9402a3f0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 -- 172.21.15.78:0/2327089543 shutdown_connections 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2327089543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0109100 0x7fcda0110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2327089543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0104980 0x7fcda01089d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2327089543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0103fd0 0x7fcda01043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 -- 172.21.15.78:0/2327089543 >> 172.21.15.78:0/2327089543 conn(0x7fcda00ffa30 msgr2=0x7fcda0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 -- 172.21.15.78:0/2327089543 shutdown_connections 2024-09-16T01:12:59.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.979+0000 7fcda64f0640 1 -- 172.21.15.78:0/2327089543 wait complete. 2024-09-16T01:12:59.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.980+0000 7fcda64f0640 1 Processor -- start 2024-09-16T01:12:59.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.980+0000 7fcda64f0640 1 -- start start 2024-09-16T01:12:59.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0103fd0 0x7fcda0073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0104980 0x7fcda0073e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcda0114150 con 0x7fcda0103fd0 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcda0113fd0 con 0x7fcda0104980 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda64f0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01142d0 con 0x7fcda0109100 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda4a8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda4a8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:60734/0 (socket says 172.21.15.78:60734) 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcda4a8d640 1 -- 172.21.15.78:0/2028116146 learned_addr learned my addr 172.21.15.78:0/2028116146 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.981+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0103fd0 0x7fcda0073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 -- 172.21.15.78:0/2028116146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0104980 msgr2=0x7fcda0073e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0104980 0x7fcda0073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0104980 0x7fcda0073e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 -- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0103fd0 msgr2=0x7fcda0073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0103fd0 0x7fcda0073910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 -- 172.21.15.78:0/2028116146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda006d3b0 con 0x7fcda0109100 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800eaa0 tx=0x7fcd9800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:59.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcd9d7fa640 1 -- 172.21.15.78:0/2028116146 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98004d20 con 0x7fcda0109100 2024-09-16T01:12:59.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcd9d7fa640 1 -- 172.21.15.78:0/2028116146 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd98005020 con 0x7fcda0109100 2024-09-16T01:12:59.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcd9d7fa640 1 -- 172.21.15.78:0/2028116146 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980116d0 con 0x7fcda0109100 2024-09-16T01:12:59.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda006d6a0 con 0x7fcda0109100 2024-09-16T01:12:59.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.982+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda0071610 con 0x7fcda0109100 2024-09-16T01:12:59.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.986+0000 7fcd9d7fa640 1 -- 172.21.15.78:0/2028116146 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fcd980051d0 con 0x7fcda0109100 2024-09-16T01:12:59.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 Processor -- start 2024-09-16T01:12:59.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcd9d7fa640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 0x7fcd880a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 -- start start 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcd9d7fa640 1 -- 172.21.15.78:0/2028116146 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fcd980d02f0 con 0x7fcda0109100 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 0x7fcd880a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b6bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6c170 0x561ea9b764f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561ea9ad0160 con 0x561ea9b76a30 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561ea9a77720 con 0x561ea9b6b850 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.987+0000 7fcda88c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561ea9acc120 con 0x561ea9b6c170 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41476/0 (socket says 172.21.15.78:41476) 2024-09-16T01:12:59.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 -- 172.21.15.78:0/3426730955 learned_addr learned my addr 172.21.15.78:0/3426730955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 -- 172.21.15.78:0/3426730955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6c170 msgr2=0x561ea9b764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6c170 0x561ea9b764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6c170 0x561ea9b764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 -- 172.21.15.78:0/3426730955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 msgr2=0x561ea9b6bc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b6bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 -- 172.21.15.78:0/3426730955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea9841ba0 con 0x561ea9b76a30 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcd98098bb0 tx=0x7fcd9809c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5dcecbf1f28992 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 0x7fcd880a58c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005c40 tx=0x7fcd94004040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.988+0000 7fcd867fc640 1 -- 172.21.15.78:0/3426730955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9801f070 con 0x561ea9b76a30 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.989+0000 7fcd867fc640 1 -- 172.21.15.78:0/3426730955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd980a4e50 con 0x561ea9b76a30 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.989+0000 7fcd867fc640 1 -- 172.21.15.78:0/3426730955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980bc0d0 con 0x561ea9b76a30 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.989+0000 7fcda88c9740 1 -- 172.21.15.78:0/3426730955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 msgr2=0x561ea9b78e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.989+0000 7fcda88c9740 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcd98098bb0 tx=0x7fcd9809c9a0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.990+0000 7fcda88c9740 1 -- 172.21.15.78:0/3426730955 shutdown_connections 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.990+0000 7fcda88c9740 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b78e20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.990+0000 7fcda88c9740 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6c170 0x561ea9b764f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.990+0000 7fcda88c9740 1 --2- 172.21.15.78:0/3426730955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b6bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.990+0000 7fcda88c9740 1 -- 172.21.15.78:0/3426730955 >> 172.21.15.78:0/3426730955 conn(0x561ea9b67de0 msgr2=0x561ea9b681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.991+0000 7fcda88c9740 1 -- 172.21.15.78:0/3426730955 shutdown_connections 2024-09-16T01:12:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.991+0000 7fcda88c9740 1 -- 172.21.15.78:0/3426730955 wait complete. 2024-09-16T01:12:59.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.992+0000 7fcda88c9740 1 -- start start 2024-09-16T01:12:59.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.992+0000 7fcda88c9740 1 Processor -- start 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6b850 0x561ea9b73950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561ea9b79600 con 0x561ea9b76a30 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561ea9a58900 con 0x561ea9b6c170 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561ea9b79780 con 0x561ea9b6b850 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59440/0 (socket says 172.21.15.78:59440) 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 -- 172.21.15.78:0/2756566562 learned_addr learned my addr 172.21.15.78:0/2756566562 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:12:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6b850 0x561ea9b73950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcda88c9740 10 client.0 _ll_register_callbacks cb 0x561ea9a56780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 -- 172.21.15.78:0/2756566562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6b850 msgr2=0x561ea9b73950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6b850 0x561ea9b73950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 -- 172.21.15.78:0/2756566562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 msgr2=0x561ea9b743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 -- 172.21.15.78:0/2756566562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea9a98c50 con 0x561ea9b6c170 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.993+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005270 tx=0x7fcd9402a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d957f610d671d47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9404e060 con 0x561ea9b6c170 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fcd94004b90 con 0x561ea9b6c170 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd980a4e50 con 0x561ea9b6c170 2024-09-16T01:12:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcd94055050 con 0x561ea9b6c170 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 10 client.25130 fetch_fsmap learned FSMap version 28 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ea9b852c0 con 0x561ea9b6c170 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd94004d90 con 0x561ea9b6c170 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fcd940038f0 con 0x561ea9b6c170 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 10 client.25130 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 10 client.25130 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 20 client.25130 populate_metadata read hostname 'smithi078' 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9b86920 0x561ea9b88d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 4 client.25130 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.994+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcd980a4e50 con 0x561ea9b86920 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.995+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9b86920 0x561ea9b88d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.995+0000 7fcd7f7fe640 10 client.25130 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:12:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.995+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9b86920 0x561ea9b88d10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=507d22174ca28ff server_cookie=5e415d6a31b3e791 in_seq=0 out_seq=0 2024-09-16T01:13:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.996+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2756566562 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fcd980a4e50 con 0x561ea9b86920 2024-09-16T01:13:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.996+0000 7fcd7f7fe640 10 client.25130 handle_command_reply: tid=0 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.996+0000 7fcda88c9740 1 client.25130 shutdown 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 msgr2=0x561ea9b73e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005270 tx=0x7fcd9402a3f0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 shutdown_connections 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9b86920 0x561ea9b88d10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcd7f7fe640 0 client.25130 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b76a30 0x561ea9b743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6c170 0x561ea9b73e90 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005270 tx=0x7fcd9402a3f0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2756566562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b6b850 0x561ea9b73950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 >> 172.21.15.78:0/2756566562 conn(0x561ea9b67de0 msgr2=0x561ea9b6dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 shutdown_connections 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.997+0000 7fcda88c9740 1 -- 172.21.15.78:0/2756566562 wait complete. 2024-09-16T01:13:00.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:12:59.998+0000 7fcda88c9740 20 client.25130 trim_cache size 0 max 16384 2024-09-16T01:13:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.024+0000 7fcda88c9740 1 Processor -- start 2024-09-16T01:13:00.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 -- start start 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b76a30 0x561ea9b8e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 0x561ea9b70bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01d1760 con 0x561ea9b6b850 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01cf000 con 0x561ea9b76a30 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda88c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01ccf50 con 0x561ea9b8eb60 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b76a30 0x561ea9b8e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41498/0 (socket says 172.21.15.78:41498) 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 -- 172.21.15.78:0/1976557897 learned_addr learned my addr 172.21.15.78:0/1976557897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 0x561ea9b70bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 -- 172.21.15.78:0/1976557897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 msgr2=0x561ea9b70bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:00.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 0x561ea9b70bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 -- 172.21.15.78:0/1976557897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b76a30 msgr2=0x561ea9b8e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b76a30 0x561ea9b8e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcd9ffff640 1 -- 172.21.15.78:0/1976557897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea9b89390 con 0x561ea9b6b850 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.025+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 0x561ea9b70bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.026+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000a980 tx=0x7fcd9000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf31977045116ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.026+0000 7fcd85ffb640 1 -- 172.21.15.78:0/1976557897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9001a330 con 0x561ea9b6b850 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.026+0000 7fcd85ffb640 1 -- 172.21.15.78:0/1976557897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd9001a9b0 con 0x561ea9b6b850 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.026+0000 7fcd85ffb640 1 -- 172.21.15.78:0/1976557897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90011840 con 0x561ea9b6b850 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 -- 172.21.15.78:0/1976557897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 msgr2=0x561ea9b759d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcd9000a980 tx=0x7fcd9000a2d0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 -- 172.21.15.78:0/1976557897 shutdown_connections 2024-09-16T01:13:00.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b8eb60 0x561ea9b70bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b76a30 0x561ea9b8e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 --2- 172.21.15.78:0/1976557897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b6b850 0x561ea9b759d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.027+0000 7fcda88c9740 1 -- 172.21.15.78:0/1976557897 >> 172.21.15.78:0/1976557897 conn(0x561ea9b67de0 msgr2=0x561ea9b72ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.028+0000 7fcda88c9740 1 -- 172.21.15.78:0/1976557897 shutdown_connections 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.028+0000 7fcda88c9740 1 -- 172.21.15.78:0/1976557897 wait complete. 2024-09-16T01:13:00.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.028+0000 7fcda88c9740 1 -- start start 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 Processor -- start 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b76a30 0x561ea9b7b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 0x561ea9b7b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01cd4c0 con 0x561ea9b8eb60 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01ce930 con 0x561ea9b6b850 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcda01cdb90 con 0x561ea9b76a30 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.029+0000 7fcda88c9740 10 client.0 _ll_register_callbacks cb 0x7fcd94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59460/0 (socket says 172.21.15.78:59460) 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 -- 172.21.15.78:0/2289626391 learned_addr learned my addr 172.21.15.78:0/2289626391 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b76a30 0x561ea9b7b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 0x561ea9b7b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 -- 172.21.15.78:0/2289626391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b76a30 msgr2=0x561ea9b7b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b76a30 0x561ea9b7b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 -- 172.21.15.78:0/2289626391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 msgr2=0x561ea9b7b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 0x561ea9b7b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 -- 172.21.15.78:0/2289626391 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea9b633f0 con 0x561ea9b6b850 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcda4a8d640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 0x561ea9b7b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd9ffff640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcd90013360 tx=0x7fcd90005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=134be1e5688918f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.030+0000 7fcd7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:13:00.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.031+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd90031060 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.031+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fcd9001a330 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.031+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd9001a9b0 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.032+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcd90038050 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 10 client.25136 fetch_fsmap learned FSMap version 28 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ea9b68f50 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd9000aba0 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fcd90023420 con 0x561ea9b6b850 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 10 client.25136 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 10 client.25136 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 20 client.25136 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.033+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9bf2670 0x561ea9bf4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.034+0000 7fcda88c9740 4 client.25136 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-16T01:13:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.034+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x7fcd9001a9b0 con 0x561ea9bf2670 2024-09-16T01:13:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.034+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9bf2670 0x561ea9bf4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.035+0000 7fcd7f7fe640 10 client.25136 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.035+0000 7fcd9f7fe640 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9bf2670 0x561ea9bf4a90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a0030f3fd5a225 server_cookie=7cf5d65e6bc41b3d in_seq=0 out_seq=0 2024-09-16T01:13:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.312+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2289626391 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7fcd9001a9b0 con 0x561ea9bf2670 2024-09-16T01:13:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.312+0000 7fcd7f7fe640 10 client.25136 handle_command_reply: tid=0 2024-09-16T01:13:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.312+0000 7fcda88c9740 1 client.25136 shutdown 2024-09-16T01:13:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.312+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 msgr2=0x561ea9b7ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.312+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcd90013360 tx=0x7fcd90005740 comp rx=0 tx=0).stop 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 shutdown_connections 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561ea9bf2670 0x561ea9bf4a90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ea9b8eb60 0x561ea9b7b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ea9b76a30 0x561ea9b7b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcd7f7fe640 0 client.25136 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 --2- 172.21.15.78:0/2289626391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ea9b6b850 0x561ea9b7ee00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 >> 172.21.15.78:0/2289626391 conn(0x561ea9b67de0 msgr2=0x561ea9b74130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 shutdown_connections 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 1 -- 172.21.15.78:0/2289626391 wait complete. 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda88c9740 20 client.25136 trim_cache size 0 max 16384 2024-09-16T01:13:00.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 28, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 6, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "d58bb50b": { 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 6, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:13:00.317 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 139.6, 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:00.318 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 msgr2=0x7fcd880a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 0x7fcd880a58c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005c40 tx=0x7fcd94004040 comp rx=0 tx=0).stop 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 msgr2=0x7fcda006cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800eaa0 tx=0x7fcd9800eea0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 shutdown_connections 2024-09-16T01:13:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fcd880a3400 0x7fcd880a58c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcda0109100 0x7fcda006cb80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcda0104980 0x7fcda0073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 --2- 172.21.15.78:0/2028116146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda0103fd0 0x7fcda0073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.313+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 >> 172.21.15.78:0/2028116146 conn(0x7fcda00ffa30 msgr2=0x7fcda0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.314+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 shutdown_connections 2024-09-16T01:13:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:00.314+0000 7fcda64f0640 1 -- 172.21.15.78:0/2028116146 wait complete. 2024-09-16T01:13:00.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 105.0 s, 2301.0 tps, lat 39.354 ms stddev 65.354 2024-09-16T01:13:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:00 smithi148 ceph-mon[35030]: pgmap v69: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 1.26k op/s 2024-09-16T01:13:01.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:00 smithi078 ceph-mon[30937]: pgmap v69: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 1.26k op/s 2024-09-16T01:13:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:00 smithi193 ceph-mon[35701]: pgmap v69: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 1.26k op/s 2024-09-16T01:13:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 -- 172.21.15.78:0/1641713660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81009d0 msgr2=0x7ffaa8100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/1641713660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81009d0 0x7ffaa8100e30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009930 tx=0x7ffa9802a420 comp rx=0 tx=0).stop 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 -- 172.21.15.78:0/1641713660 shutdown_connections 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/1641713660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 0x7ffaa810d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/1641713660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81009d0 0x7ffaa8100e30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/1641713660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81069e0 0x7ffaa8106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 -- 172.21.15.78:0/1641713660 >> 172.21.15.78:0/1641713660 conn(0x7ffaa80fc6b0 msgr2=0x7ffaa80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 -- 172.21.15.78:0/1641713660 shutdown_connections 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.589+0000 7ffaaf48d640 1 -- 172.21.15.78:0/1641713660 wait complete. 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 Processor -- start 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 -- start start 2024-09-16T01:13:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81009d0 0x7ffaa810e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 0x7ffaa810e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8111ed0 con 0x7ffaa81069e0 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8111d50 con 0x7ffaa8101370 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaaf48d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa8112050 con 0x7ffaa81009d0 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaada2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaada2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41526/0 (socket says 172.21.15.78:41526) 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.590+0000 7ffaada2a640 1 -- 172.21.15.78:0/559810150 learned_addr learned my addr 172.21.15.78:0/559810150 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaaca28640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 0x7ffaa810e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaad229640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81009d0 0x7ffaa810e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 -- 172.21.15.78:0/559810150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81009d0 msgr2=0x7ffaa810e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81009d0 0x7ffaa810e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 -- 172.21.15.78:0/559810150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 msgr2=0x7ffaa810e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 0x7ffaa810e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 -- 172.21.15.78:0/559810150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa81a68c0 con 0x7ffaa81069e0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaada2a640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000eaa0 tx=0x7ffaa000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffa9e7fc640 1 -- 172.21.15.78:0/559810150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000cdb0 con 0x7ffaa81069e0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffa9e7fc640 1 -- 172.21.15.78:0/559810150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa0005020 con 0x7ffaa81069e0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa81a6bb0 con 0x7ffaa81069e0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffa9e7fc640 1 -- 172.21.15.78:0/559810150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00116b0 con 0x7ffaa81069e0 2024-09-16T01:13:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.591+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa81a7080 con 0x7ffaa81069e0 2024-09-16T01:13:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.594+0000 7ffa9e7fc640 1 -- 172.21.15.78:0/559810150 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7ffaa0005560 con 0x7ffaa81069e0 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.594+0000 7ffab222d740 1 Processor -- start 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.594+0000 7ffab222d740 1 -- start start 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a0af0 0x562bc11a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562bc11052f0 con 0x562bc11a1410 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562bc10ad270 con 0x562bc11abcb0 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffab222d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562bc11012b0 con 0x562bc11a0af0 2024-09-16T01:13:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffa9e7fc640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 0x7ffa940a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41530/0 (socket says 172.21.15.78:41530) 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 -- 172.21.15.78:0/4210283007 learned_addr learned my addr 172.21.15.78:0/4210283007 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaaca28640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a0af0 0x562bc11a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaada2a640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 0x7ffa940a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaaca28640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 -- 172.21.15.78:0/4210283007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a0af0 msgr2=0x562bc11a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a0af0 0x562bc11a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 -- 172.21.15.78:0/4210283007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 msgr2=0x562bc11ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 -- 172.21.15.78:0/4210283007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bc10ac340 con 0x562bc11a1410 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffa9e7fc640 1 -- 172.21.15.78:0/559810150 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffaa00d02b0 con 0x7ffaa81069e0 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaaca28640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaad229640 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400aba0 tx=0x7ffaa400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfd0b515df36d3a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffaada2a640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 0x7ffa940a5920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0098c20 tx=0x7ffaa00bc040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.595+0000 7ffa937fe640 1 -- 172.21.15.78:0/4210283007 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa401d070 con 0x562bc11a1410 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffa937fe640 1 -- 172.21.15.78:0/4210283007 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa4018b20 con 0x562bc11a1410 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffa937fe640 1 -- 172.21.15.78:0/4210283007 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4010af0 con 0x562bc11a1410 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 -- 172.21.15.78:0/4210283007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 msgr2=0x562bc11ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400aba0 tx=0x7ffaa400f790 comp rx=0 tx=0).stop 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 -- 172.21.15.78:0/4210283007 shutdown_connections 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a1410 0x562bc11ab770 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 --2- 172.21.15.78:0/4210283007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a0af0 0x562bc11a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 -- 172.21.15.78:0/4210283007 >> 172.21.15.78:0/4210283007 conn(0x562bc119d060 msgr2=0x562bc119d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 -- 172.21.15.78:0/4210283007 shutdown_connections 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.596+0000 7ffab222d740 1 -- 172.21.15.78:0/4210283007 wait complete. 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 -- start start 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 Processor -- start 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562bc11ae880 con 0x562bc11a0af0 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562bc108e290 con 0x562bc11abcb0 2024-09-16T01:13:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562bc11aea00 con 0x562bc11a1410 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffab222d740 10 client.0 _ll_register_callbacks cb 0x562bc10bf140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41542/0 (socket says 172.21.15.78:41542) 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffaaca28640 1 -- 172.21.15.78:0/1508453733 learned_addr learned my addr 172.21.15.78:0/1508453733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffaad229640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.597+0000 7ffaaca28640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 -- 172.21.15.78:0/1508453733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 msgr2=0x562bc11a9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 -- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 msgr2=0x562bc11a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 -- 172.21.15.78:0/1508453733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bc0e18210 con 0x562bc11a0af0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffaaca28640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ffa9802a940 tx=0x7ffa98005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0015b83f23c766b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffa90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98035040 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa4018b20 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffa98005020 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa98051050 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffab222d740 10 client.15666 fetch_fsmap learned FSMap version 28 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562bc0ee4620 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.598+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa4018b20 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ffa98049020 con 0x562bc11a0af0 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 10 client.15666 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 20 client.15666 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc11bbbe0 0x562bc11bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 4 client.15666 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bc1112a20 con 0x562bc11bbbe0 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffaad229640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc11bbbe0 0x562bc11bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffaad229640 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc11bbbe0 0x562bc11bdfd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1dd3be39f9b7de5 server_cookie=d62e95323fd36449 in_seq=0 out_seq=0 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.599+0000 7ffa90ff9640 10 client.15666 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffa90ff9640 1 -- 172.21.15.78:0/1508453733 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562bc1112a20 con 0x562bc11bbbe0 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffa90ff9640 10 client.15666 handle_command_reply: tid=0 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 client.15666 shutdown 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 msgr2=0x562bc11a8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ffa9802a940 tx=0x7ffa98005e40 comp rx=0 tx=0).stop 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 shutdown_connections 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc11bbbe0 0x562bc11bdfd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 --2- 172.21.15.78:0/1508453733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a8c10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffa90ff9640 0 client.15666 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 >> 172.21.15.78:0/1508453733 conn(0x562bc119d060 msgr2=0x562bc119c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 shutdown_connections 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.600+0000 7ffab222d740 1 -- 172.21.15.78:0/1508453733 wait complete. 2024-09-16T01:13:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.601+0000 7ffab222d740 20 client.15666 trim_cache size 0 max 16384 2024-09-16T01:13:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.615+0000 7ffab222d740 1 Processor -- start 2024-09-16T01:13:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.615+0000 7ffab222d740 1 -- start start 2024-09-16T01:13:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81b3ae0 con 0x562bc11a0af0 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81b1380 con 0x562bc11abcb0 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffab222d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81af1d0 con 0x562bc11a1410 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41558/0 (socket says 172.21.15.78:41558) 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 -- 172.21.15.78:0/299182544 learned_addr learned my addr 172.21.15.78:0/299182544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 -- 172.21.15.78:0/299182544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 msgr2=0x562bc11c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 -- 172.21.15.78:0/299182544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 msgr2=0x562bc11c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 -- 172.21.15.78:0/299182544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bc11972f0 con 0x562bc11a0af0 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffaaca28640 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa98005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4477f92b3227e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffa92ffd640 1 -- 172.21.15.78:0/299182544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98041040 con 0x562bc11a0af0 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffa92ffd640 1 -- 172.21.15.78:0/299182544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa98004c50 con 0x562bc11a0af0 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.616+0000 7ffa92ffd640 1 -- 172.21.15.78:0/299182544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98005000 con 0x562bc11a0af0 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- 172.21.15.78:0/299182544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 msgr2=0x562bc11a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ffa98000c00 tx=0x7ffa98005d50 comp rx=0 tx=0).stop 2024-09-16T01:13:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- 172.21.15.78:0/299182544 shutdown_connections 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11abcb0 0x562bc11c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11c38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 --2- 172.21.15.78:0/299182544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11a0af0 0x562bc11a94c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- 172.21.15.78:0/299182544 >> 172.21.15.78:0/299182544 conn(0x562bc119d060 msgr2=0x562bc11aac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- 172.21.15.78:0/299182544 shutdown_connections 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- 172.21.15.78:0/299182544 wait complete. 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 -- start start 2024-09-16T01:13:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.617+0000 7ffab222d740 1 Processor -- start 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11a0af0 0x562bc11b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81af840 con 0x562bc11abcb0 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81b0cb0 con 0x562bc11a0af0 2024-09-16T01:13:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa81aff10 con 0x562bc11a1410 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffab222d740 10 client.0 _ll_register_callbacks cb 0x7ffa98005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41564/0 (socket says 172.21.15.78:41564) 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 -- 172.21.15.78:0/2191759981 learned_addr learned my addr 172.21.15.78:0/2191759981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaad229640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 -- 172.21.15.78:0/2191759981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 msgr2=0x562bc11a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 -- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11a0af0 msgr2=0x562bc11b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11a0af0 0x562bc11b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 -- 172.21.15.78:0/2191759981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bc1225890 con 0x562bc11abcb0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffaaca28640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009660 tx=0x7ffa98004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740b8ebdf5643c33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.618+0000 7ffa90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98035070 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffa980316c0 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa98004c50 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa980318c0 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 10 client.15672 fetch_fsmap learned FSMap version 28 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562bc1226a40 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa98004c50 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ffa98069030 con 0x562bc11abcb0 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 10 client.15672 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 20 client.15672 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc1227040 0x562bc1229460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 4 client.15672 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.619+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562bc110cd70 con 0x562bc1227040 2024-09-16T01:13:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.620+0000 7ffaad229640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc1227040 0x562bc1229460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.620+0000 7ffaad229640 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc1227040 0x562bc1229460 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3d058c437da074 server_cookie=36408dc7b7370ec3 in_seq=0 out_seq=0 2024-09-16T01:13:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.620+0000 7ffa90ff9640 10 client.15672 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:02.784 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffa90ff9640 1 -- 172.21.15.78:0/2191759981 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562bc110cd70 con 0x562bc1227040 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffa90ff9640 10 client.15672 handle_command_reply: tid=0 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 client.15672 shutdown 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 msgr2=0x562bc11a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffa98009660 tx=0x7ffa98004e80 comp rx=0 tx=0).stop 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 shutdown_connections 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562bc1227040 0x562bc1229460 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562bc11abcb0 0x562bc11a61c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562bc11a1410 0x562bc11a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffa90ff9640 0 client.15672 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:02.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 --2- 172.21.15.78:0/2191759981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562bc11a0af0 0x562bc11b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 >> 172.21.15.78:0/2191759981 conn(0x562bc119d060 msgr2=0x562bc11a2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 shutdown_connections 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 1 -- 172.21.15.78:0/2191759981 wait complete. 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.782+0000 7ffab222d740 20 client.15672 trim_cache size 0 max 16384 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:13:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:13:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 msgr2=0x7ffa940a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 0x7ffa940a5920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0098c20 tx=0x7ffaa00bc040 comp rx=0 tx=0).stop 2024-09-16T01:13:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 msgr2=0x7ffaa810edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000eaa0 tx=0x7ffaa000ef60 comp rx=0 tx=0).stop 2024-09-16T01:13:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 shutdown_connections 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffa940a3460 0x7ffa940a5920 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaa81069e0 0x7ffaa810edb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffaa8101370 0x7ffaa810e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 --2- 172.21.15.78:0/559810150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaa81009d0 0x7ffaa810e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 >> 172.21.15.78:0/559810150 conn(0x7ffaa80fc6b0 msgr2=0x7ffaa80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 shutdown_connections 2024-09-16T01:13:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:02.783+0000 7ffaaf48d640 1 -- 172.21.15.78:0/559810150 wait complete. 2024-09-16T01:13:03.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:02 smithi148 ceph-mon[35030]: pgmap v70: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 970 op/s 2024-09-16T01:13:03.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:02 smithi078 ceph-mon[30937]: pgmap v70: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 970 op/s 2024-09-16T01:13:03.279 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd58bb50b', quiesce took 7.9 seconds. Will release after: 11.65 2024-09-16T01:13:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:02 smithi193 ceph-mon[35701]: pgmap v70: 129 pgs: 129 active+clean; 1.4 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 MiB/s wr, 970 op/s 2024-09-16T01:13:04.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:03 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:04.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:03 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:04.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:03 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:04 smithi148 ceph-mon[35030]: pgmap v71: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-09-16T01:13:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:04 smithi078 ceph-mon[30937]: pgmap v71: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-09-16T01:13:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:04 smithi193 ceph-mon[35701]: pgmap v71: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 MiB/s wr, 1.07k op/s 2024-09-16T01:13:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 110.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:13:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:06 smithi148 ceph-mon[35030]: pgmap v72: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 861 op/s 2024-09-16T01:13:07.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:06 smithi078 ceph-mon[30937]: pgmap v72: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 861 op/s 2024-09-16T01:13:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:06 smithi193 ceph-mon[35701]: pgmap v72: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 861 op/s 2024-09-16T01:13:09.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:08 smithi148 ceph-mon[35030]: pgmap v73: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 805 op/s 2024-09-16T01:13:09.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:08 smithi078 ceph-mon[30937]: pgmap v73: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 805 op/s 2024-09-16T01:13:09.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:08 smithi193 ceph-mon[35701]: pgmap v73: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 805 op/s 2024-09-16T01:13:10.941 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 115.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:13:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:10 smithi148 ceph-mon[35030]: pgmap v74: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 808 op/s 2024-09-16T01:13:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:10 smithi078 ceph-mon[30937]: pgmap v74: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 808 op/s 2024-09-16T01:13:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:10 smithi193 ceph-mon[35701]: pgmap v74: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 808 op/s 2024-09-16T01:13:13.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:12 smithi148 ceph-mon[35030]: pgmap v75: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1024 KiB/s wr, 129 op/s 2024-09-16T01:13:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:12 smithi078 ceph-mon[30937]: pgmap v75: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1024 KiB/s wr, 129 op/s 2024-09-16T01:13:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:12 smithi193 ceph-mon[35701]: pgmap v75: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1024 KiB/s wr, 129 op/s 2024-09-16T01:13:14.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:13:15.198 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:14 smithi078 ceph-mon[30937]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 318 op/s 2024-09-16T01:13:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:14 smithi193 ceph-mon[35701]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 318 op/s 2024-09-16T01:13:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:14 smithi148 ceph-mon[35030]: pgmap v76: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 318 op/s 2024-09-16T01:13:15.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 -- 172.21.15.78:0/843337670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4106a80 msgr2=0x7f94a4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/843337670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4106a80 0x7f94a4106e60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9498009930 tx=0x7f949802a420 comp rx=0 tx=0).stop 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 -- 172.21.15.78:0/843337670 shutdown_connections 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/843337670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4101420 0x7f94a410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/843337670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4100a80 0x7f94a4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/843337670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4106a80 0x7f94a4106e60 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 -- 172.21.15.78:0/843337670 >> 172.21.15.78:0/843337670 conn(0x7f94a40fc7a0 msgr2=0x7f94a40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 -- 172.21.15.78:0/843337670 shutdown_connections 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.705+0000 7f94ac6bc640 1 -- 172.21.15.78:0/843337670 wait complete. 2024-09-16T01:13:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 Processor -- start 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 -- start start 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 0x7f94a4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 0x7f94a419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110ea0 con 0x7f94a4100a80 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110d20 con 0x7f94a4106a80 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94ac6bc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4111020 con 0x7f94a4101420 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34860/0 (socket says 172.21.15.78:34860) 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 -- 172.21.15.78:0/748071564 learned_addr learned my addr 172.21.15.78:0/748071564 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 -- 172.21.15.78:0/748071564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 msgr2=0x7f94a4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94a9c57640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 0x7f94a4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 0x7f94a4198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aac59640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 0x7f94a419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 -- 172.21.15.78:0/748071564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 msgr2=0x7f94a419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 0x7f94a419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 -- 172.21.15.78:0/748071564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a419d3d0 con 0x7f94a4100a80 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94a9c57640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 0x7f94a4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aac59640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 0x7f94a419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.706+0000 7f94aa458640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9498009a50 tx=0x7f9498004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.707+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949803f030 con 0x7f94a4100a80 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.707+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9498003ce0 con 0x7f94a4100a80 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.707+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a419d660 con 0x7f94a4100a80 2024-09-16T01:13:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.707+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498035070 con 0x7f94a4100a80 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.707+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a41a4fa0 con 0x7f94a4100a80 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.708+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a41021c0 con 0x7f94a4100a80 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.709+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f9498042030 con 0x7f94a4100a80 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.709+0000 7f94977fe640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 0x7f94900a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.710+0000 7f94a9c57640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 0x7f94900a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.710+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f94980f0b50 con 0x7f94a4100a80 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.710+0000 7f94a9c57640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 0x7f94900a5870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f94a0006fd0 tx=0x7f94a0008040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.713+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f94980bd1a0 con 0x7f94a4100a80 2024-09-16T01:13:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 -- 172.21.15.78:0/2550357301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08501069c0 msgr2=0x7f0850106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 --2- 172.21.15.78:0/2550357301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08501069c0 0x7f0850106da0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0840009930 tx=0x7f084002a420 comp rx=0 tx=0).stop 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 -- 172.21.15.78:0/2550357301 shutdown_connections 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 --2- 172.21.15.78:0/2550357301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0850101350 0x7f085010d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 --2- 172.21.15.78:0/2550357301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501009b0 0x7f0850100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 --2- 172.21.15.78:0/2550357301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08501069c0 0x7f0850106da0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 -- 172.21.15.78:0/2550357301 >> 172.21.15.78:0/2550357301 conn(0x7f0850077990 msgr2=0x7f08500fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 -- 172.21.15.78:0/2550357301 shutdown_connections 2024-09-16T01:13:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.874+0000 7f0854f9f640 1 -- 172.21.15.78:0/2550357301 wait complete. 2024-09-16T01:13:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 Processor -- start 2024-09-16T01:13:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 -- start start 2024-09-16T01:13:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08501009b0 0x7f0850198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501069c0 0x7f085019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0850110e00 con 0x7f0850101350 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0850110c80 con 0x7f08501009b0 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f0854f9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0850110f80 con 0x7f08501069c0 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f084dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f084dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34882/0 (socket says 172.21.15.78:34882) 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.875+0000 7f084dd9b640 1 -- 172.21.15.78:0/3914651800 learned_addr learned my addr 172.21.15.78:0/3914651800 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084e59c640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08501009b0 0x7f0850198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084ed9d640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501069c0 0x7f085019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 -- 172.21.15.78:0/3914651800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501069c0 msgr2=0x7f085019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501069c0 0x7f085019cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 -- 172.21.15.78:0/3914651800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08501009b0 msgr2=0x7f0850198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08501009b0 0x7f0850198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 -- 172.21.15.78:0/3914651800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085019d330 con 0x7f0850101350 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f084dd9b640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f084400d950 tx=0x7f084400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f083f7fe640 1 -- 172.21.15.78:0/3914651800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0844017070 con 0x7f0850101350 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085019d620 con 0x7f0850101350 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f083f7fe640 1 -- 172.21.15.78:0/3914651800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0844009c30 con 0x7f0850101350 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f083f7fe640 1 -- 172.21.15.78:0/3914651800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084400f6b0 con 0x7f0850101350 2024-09-16T01:13:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.876+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08501a4eb0 con 0x7f0850101350 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f083f7fe640 1 -- 172.21.15.78:0/3914651800 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f08440109d0 con 0x7f0850101350 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 Processor -- start 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 -- start start 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 0x559fd56e1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559fd563b2f0 con 0x559fd56d6af0 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559fd55e3270 con 0x559fd56d7410 2024-09-16T01:13:15.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f0857e16740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559fd56372b0 con 0x559fd56e1cb0 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f084ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f084ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34896/0 (socket says 172.21.15.78:34896) 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f084ed9d640 1 -- 172.21.15.78:0/307019118 learned_addr learned my addr 172.21.15.78:0/307019118 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f083f7fe640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 0x7f08380a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.879+0000 7f084e59c640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 0x559fd56e1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084ed9d640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 0x7f08380a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f083f7fe640 1 -- 172.21.15.78:0/3914651800 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f08440cfc50 con 0x7f0850101350 2024-09-16T01:13:15.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084dd9b640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 -- 172.21.15.78:0/307019118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 msgr2=0x559fd56e40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 -- 172.21.15.78:0/307019118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 msgr2=0x559fd56d6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 -- 172.21.15.78:0/307019118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd55e2340 con 0x559fd56d7410 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084ed9d640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084e59c640 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 0x559fd56e1770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0840005940 tx=0x7f0840005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=179956ddc9c71008 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f084ed9d640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 0x7f08380a5920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0848005f40 tx=0x7f0848004ab0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f083cff9640 1 -- 172.21.15.78:0/307019118 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840040070 con 0x559fd56d7410 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f083cff9640 1 -- 172.21.15.78:0/307019118 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08400319f0 con 0x559fd56d7410 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.880+0000 7f083cff9640 1 -- 172.21.15.78:0/307019118 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840035040 con 0x559fd56d7410 2024-09-16T01:13:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- 172.21.15.78:0/307019118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 msgr2=0x559fd56e1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 0x559fd56e1770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0840005940 tx=0x7f0840005c00 comp rx=0 tx=0).stop 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- 172.21.15.78:0/307019118 shutdown_connections 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d7410 0x559fd56e1770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 --2- 172.21.15.78:0/307019118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56d6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- 172.21.15.78:0/307019118 >> 172.21.15.78:0/307019118 conn(0x559fd56d3060 msgr2=0x559fd56d3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- 172.21.15.78:0/307019118 shutdown_connections 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- 172.21.15.78:0/307019118 wait complete. 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.881+0000 7f0857e16740 1 -- start start 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 Processor -- start 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56def90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56df4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559fd56e4880 con 0x559fd56d6af0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559fd55c4290 con 0x559fd56e1cb0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559fd56e4a00 con 0x559fd56d7410 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0857e16740 10 client.0 _ll_register_callbacks cb 0x559fd55f5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34902/0 (socket says 172.21.15.78:34902) 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 -- 172.21.15.78:0/1714547333 learned_addr learned my addr 172.21.15.78:0/1714547333 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084ed9d640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56df4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084dd9b640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56def90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 -- 172.21.15.78:0/1714547333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 msgr2=0x559fd56def90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56def90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 -- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 msgr2=0x559fd56df4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56df4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 -- 172.21.15.78:0/1714547333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd534e210 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f084e59c640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0840005280 tx=0x7f0840005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923fc3c8e7a0d4a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.882+0000 7f0835ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840040070 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08400319f0 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f08400317f0 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0840056050 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 10 client.15690 fetch_fsmap learned FSMap version 29 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fd541a620 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08400319f0 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f08400035f0 con 0x559fd56d6af0 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 10 client.15690 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T01:13:15.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 20 client.15690 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd56f18d0 0x559fd56f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 4 client.15690 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fd5648a20 con 0x559fd56f18d0 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.883+0000 7f084dd9b640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd56f18d0 0x559fd56f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.884+0000 7f0835ffb640 10 client.15690 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.884+0000 7f084dd9b640 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd56f18d0 0x559fd56f3cc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139ba98552bea6db server_cookie=7fa87f419fd55a45 in_seq=0 out_seq=0 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0835ffb640 1 -- 172.21.15.78:0/1714547333 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559fd5648a20 con 0x559fd56f18d0 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0835ffb640 10 client.15690 handle_command_reply: tid=0 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 client.15690 shutdown 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 msgr2=0x559fd56dea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0840005280 tx=0x7f0840005e80 comp rx=0 tx=0).stop 2024-09-16T01:13:15.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 shutdown_connections 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd56f18d0 0x559fd56f3cc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56df4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0835ffb640 0 client.15690 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56def90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 --2- 172.21.15.78:0/1714547333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56dea50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 >> 172.21.15.78:0/1714547333 conn(0x559fd56d3060 msgr2=0x559fd56d8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 shutdown_connections 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 1 -- 172.21.15.78:0/1714547333 wait complete. 2024-09-16T01:13:15.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.885+0000 7f0857e16740 20 client.15690 trim_cache size 0 max 16384 2024-09-16T01:13:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 Processor -- start 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 -- start start 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d6af0 0x559fd56f9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08501b1860 con 0x559fd56d7410 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f08501af100 con 0x559fd56d6af0 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f0857e16740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08501acf50 con 0x559fd56e1cb0 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f084dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f084dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34916/0 (socket says 172.21.15.78:34916) 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.913+0000 7f084dd9b640 1 -- 172.21.15.78:0/364324956 learned_addr learned my addr 172.21.15.78:0/364324956 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084e59c640 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d6af0 0x559fd56f9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 -- 172.21.15.78:0/364324956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 msgr2=0x559fd56e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 -- 172.21.15.78:0/364324956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d6af0 msgr2=0x559fd56f9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d6af0 0x559fd56f9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 -- 172.21.15.78:0/364324956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd56d7240 con 0x559fd56d7410 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f084dd9b640 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0844098620 tx=0x7f08440d2160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4567a6929a346ba3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f0837fff640 1 -- 172.21.15.78:0/364324956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0844020040 con 0x559fd56d7410 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f0837fff640 1 -- 172.21.15.78:0/364324956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0844099f50 con 0x559fd56d7410 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.914+0000 7f0837fff640 1 -- 172.21.15.78:0/364324956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084409b300 con 0x559fd56d7410 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- 172.21.15.78:0/364324956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 msgr2=0x559fd56f97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0844098620 tx=0x7f08440d2160 comp rx=0 tx=0).stop 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- 172.21.15.78:0/364324956 shutdown_connections 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56e1cb0 0x559fd56e30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d7410 0x559fd56f97a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- 172.21.15.78:0/364324956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56d6af0 0x559fd56f9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- 172.21.15.78:0/364324956 >> 172.21.15.78:0/364324956 conn(0x559fd56d3060 msgr2=0x559fd56e0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- 172.21.15.78:0/364324956 shutdown_connections 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- 172.21.15.78:0/364324956 wait complete. 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- start start 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 Processor -- start 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08501ad5c0 con 0x559fd56d6af0 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f08501aea30 con 0x559fd56e1cb0 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08501adc90 con 0x559fd56d7410 2024-09-16T01:13:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f0857e16740 10 client.0 _ll_register_callbacks cb 0x7f0840005720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.915+0000 7f084e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34922/0 (socket says 172.21.15.78:34922) 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 -- 172.21.15.78:0/2113219889 learned_addr learned my addr 172.21.15.78:0/2113219889 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 -- 172.21.15.78:0/2113219889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 msgr2=0x559fd56e54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 -- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 msgr2=0x559fd56e59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084ed9d640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56e59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 -- 172.21.15.78:0/2113219889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd56e5f20 con 0x559fd56d6af0 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084ed9d640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56e59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f084e59c640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0840006010 tx=0x7f0840003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55d7e6d84bbff80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0835ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0840040070 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f08400317f0 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0844099f50 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0840035030 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0857e16740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fd575bf90 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.916+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0844099f50 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0840004240 con 0x559fd56d6af0 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 20 client.15696 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd575c5f0 0x559fd575ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 4 client.15696 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:13:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559fd5642d70 con 0x559fd575c5f0 2024-09-16T01:13:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.917+0000 7f084dd9b640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd575c5f0 0x559fd575ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.918+0000 7f084dd9b640 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd575c5f0 0x559fd575ea10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b160ceb3cfc49f2b server_cookie=75c2f5c9ce232b6e in_seq=0 out_seq=0 2024-09-16T01:13:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.918+0000 7f0835ffb640 10 client.15696 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.924+0000 7f0835ffb640 1 -- 172.21.15.78:0/2113219889 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559fd5642d70 con 0x559fd575c5f0 2024-09-16T01:13:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.924+0000 7f0835ffb640 10 client.15696 handle_command_reply: tid=0 2024-09-16T01:13:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.924+0000 7f0857e16740 1 client.15696 shutdown 2024-09-16T01:13:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 msgr2=0x559fd56ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0840006010 tx=0x7f0840003df0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 shutdown_connections 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559fd575c5f0 0x559fd575ea10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559fd56e1cb0 0x559fd56e59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fd56d7410 0x559fd56e54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 --2- 172.21.15.78:0/2113219889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559fd56d6af0 0x559fd56ea130 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 >> 172.21.15.78:0/2113219889 conn(0x559fd56d3060 msgr2=0x559fd56e02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0835ffb640 0 client.15696 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 shutdown_connections 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 1 -- 172.21.15.78:0/2113219889 wait complete. 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0857e16740 20 client.15696 trim_cache size 0 max 16384 2024-09-16T01:13:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:13:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:13:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:13:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:13:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 msgr2=0x7f08380a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 0x7f08380a5920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0848005f40 tx=0x7f0848004ab0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 msgr2=0x7f0850198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f084400d950 tx=0x7f084400de10 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 shutdown_connections 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08380a3460 0x7f08380a5920 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08501069c0 0x7f085019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0850101350 0x7f0850198880 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 --2- 172.21.15.78:0/3914651800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08501009b0 0x7f0850198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.925+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 >> 172.21.15.78:0/3914651800 conn(0x7f0850077990 msgr2=0x7f08500ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.926+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 shutdown_connections 2024-09-16T01:13:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:15.926+0000 7f0854f9f640 1 -- 172.21.15.78:0/3914651800 wait complete. 2024-09-16T01:13:15.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 120.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:13:15.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.940+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94a4100ee0 con 0x7f94a4100a80 2024-09-16T01:13:15.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.941+0000 7f94977fe640 1 -- 172.21.15.78:0/748071564 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11920 (secure 0 0 0) 0x7f94a4100ee0 con 0x7f94a4100a80 2024-09-16T01:13:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:13:15.942 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":29,"btime":"2024-09-16T01:13:02:855468+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:13:02.780961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:13:15.942 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 29 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 msgr2=0x7f94900a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 0x7f94900a5870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f94a0006fd0 tx=0x7f94a0008040 comp rx=0 tx=0).stop 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 msgr2=0x7f94a4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9498009a50 tx=0x7f9498004270 comp rx=0 tx=0).stop 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 shutdown_connections 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f94900a33b0 0x7f94900a5870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94a4106a80 0x7f94a419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94a4101420 0x7f94a4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 --2- 172.21.15.78:0/748071564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94a4100a80 0x7f94a4198420 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:15.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 >> 172.21.15.78:0/748071564 conn(0x7f94a40fc7a0 msgr2=0x7f94a406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:15.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 shutdown_connections 2024-09-16T01:13:15.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:15.943+0000 7f94ac6bc640 1 -- 172.21.15.78:0/748071564 wait complete. 2024-09-16T01:13:16.765 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id d58bb50b --release --await' 2024-09-16T01:13:16.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id d58bb50b --release --await 2024-09-16T01:13:17.024 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:16 smithi078 ceph-mon[30937]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 234 op/s 2024-09-16T01:13:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:16 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/748071564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:16 smithi193 ceph-mon[35701]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 234 op/s 2024-09-16T01:13:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:16 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/748071564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:16 smithi148 ceph-mon[35030]: pgmap v77: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 234 op/s 2024-09-16T01:13:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:16 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/748071564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:17.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 -- 172.21.15.78:0/167226297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538102570 msgr2=0x7f35381029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 --2- 172.21.15.78:0/167226297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538102570 0x7f35381029b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3528009900 tx=0x7f352802a480 comp rx=0 tx=0).stop 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 -- 172.21.15.78:0/167226297 shutdown_connections 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 --2- 172.21.15.78:0/167226297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102ef0 0x7f353810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 --2- 172.21.15.78:0/167226297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538102570 0x7f35381029b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 --2- 172.21.15.78:0/167226297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538100e20 0x7f3538101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 -- 172.21.15.78:0/167226297 >> 172.21.15.78:0/167226297 conn(0x7f35380fc7d0 msgr2=0x7f35380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 -- 172.21.15.78:0/167226297 shutdown_connections 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 -- 172.21.15.78:0/167226297 wait complete. 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.526+0000 7f353f7f1640 1 Processor -- start 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 -- start start 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102570 0x7f3538198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538102ef0 0x7f353819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3538110fc0 con 0x7f3538100e20 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3538110e40 con 0x7f3538102ef0 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353f7f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3538111140 con 0x7f3538102570 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34952/0 (socket says 172.21.15.78:34952) 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 -- 172.21.15.78:0/1053416728 learned_addr learned my addr 172.21.15.78:0/1053416728 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538102ef0 0x7f353819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102570 0x7f3538198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 -- 172.21.15.78:0/1053416728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102570 msgr2=0x7f3538198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102570 0x7f3538198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 -- 172.21.15.78:0/1053416728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538102ef0 msgr2=0x7f353819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538102ef0 0x7f353819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 -- 172.21.15.78:0/1053416728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353819d4d0 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.527+0000 7f353d58d640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f353400b4a0 tx=0x7f353400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.528+0000 7f352e7fc640 1 -- 172.21.15.78:0/1053416728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534014070 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.528+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353819d7c0 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.528+0000 7f352e7fc640 1 -- 172.21.15.78:0/1053416728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3534010af0 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.528+0000 7f352e7fc640 1 -- 172.21.15.78:0/1053416728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534017d10 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.528+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381a50a0 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f352e7fc640 1 -- 172.21.15.78:0/1053416728 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3534017070 con 0x7f3538100e20 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 Processor -- start 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 -- start start 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c465d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c3c0620 con 0x55dc2c466240 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c367e50 con 0x55dc2c45b980 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.530+0000 7f3541bca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c3bc5e0 con 0x55dc2c45b060 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f352e7fc640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 0x7f35240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 0x7f35240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f352e7fc640 1 -- 172.21.15.78:0/1053416728 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f35340cf370 con 0x7f3538100e20 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34966/0 (socket says 172.21.15.78:34966) 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 -- 172.21.15.78:0/1579886656 learned_addr learned my addr 172.21.15.78:0/1579886656 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353d58d640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c465d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 -- 172.21.15.78:0/1579886656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 msgr2=0x55dc2c45b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 -- 172.21.15.78:0/1579886656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 msgr2=0x55dc2c465d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c465d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 -- 172.21.15.78:0/1579886656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2c358d40 con 0x55dc2c466240 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353d58d640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c465d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 0x7f35240a5870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3530005fd0 tx=0x7f3530009de0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35280314e0 tx=0x7f35280040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd8c9189b956ea3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f35237fe640 1 -- 172.21.15.78:0/1579886656 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528035030 con 0x55dc2c466240 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f35237fe640 1 -- 172.21.15.78:0/1579886656 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3528005520 con 0x55dc2c466240 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.531+0000 7f35237fe640 1 -- 172.21.15.78:0/1579886656 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528003850 con 0x55dc2c466240 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 -- 172.21.15.78:0/1579886656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 msgr2=0x55dc2c468630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f35280314e0 tx=0x7f35280040f0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 -- 172.21.15.78:0/1579886656 shutdown_connections 2024-09-16T01:13:17.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c468630 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c465d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 --2- 172.21.15.78:0/1579886656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 -- 172.21.15.78:0/1579886656 >> 172.21.15.78:0/1579886656 conn(0x55dc2c4575f0 msgr2=0x55dc2c4579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 -- 172.21.15.78:0/1579886656 shutdown_connections 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.532+0000 7f3541bca740 1 -- 172.21.15.78:0/1579886656 wait complete. 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 -- start start 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 Processor -- start 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c4630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c463b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c468e10 con 0x55dc2c45b980 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c348f10 con 0x55dc2c466240 2024-09-16T01:13:17.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc2c468f90 con 0x55dc2c45b060 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f3541bca740 10 client.0 _ll_register_callbacks cb 0x55dc2c36cdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f353cd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f353cd8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34970/0 (socket says 172.21.15.78:34970) 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f353cd8c640 1 -- 172.21.15.78:0/2491165043 learned_addr learned my addr 172.21.15.78:0/2491165043 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f353dd8e640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c463b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.533+0000 7f353d58d640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c4630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 -- 172.21.15.78:0/2491165043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 msgr2=0x55dc2c4630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c4630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 -- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 msgr2=0x55dc2c463b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c463b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 -- 172.21.15.78:0/2491165043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2c0cf580 con 0x55dc2c45b980 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f353cd8c640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3528005940 tx=0x7f35280048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef8b8af9a6a9605 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528035030 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3528005020 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3528005520 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3528005220 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 10 client.15708 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc2c199960 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3528005520 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f3528031700 con 0x55dc2c45b980 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 10 client.15708 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 10 client.15708 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.534+0000 7f3541bca740 20 client.15708 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c475bf0 0x55dc2c477fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f3541bca740 4 client.15708 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc2c3cdb90 con 0x55dc2c475bf0 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f353d58d640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c475bf0 0x55dc2c477fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f3520ff9640 10 client.15708 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.535+0000 7f353d58d640 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c475bf0 0x55dc2c477fe0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b1c772e27390578 server_cookie=4d927470ddecd1d5 in_seq=0 out_seq=0 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3520ff9640 1 -- 172.21.15.78:0/2491165043 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc2c3cdb90 con 0x55dc2c475bf0 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3520ff9640 10 client.15708 handle_command_reply: tid=0 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 client.15708 shutdown 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 msgr2=0x55dc2c463620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3528005940 tx=0x7f35280048f0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 shutdown_connections 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c475bf0 0x55dc2c477fe0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c463b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b980 0x55dc2c463620 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 --2- 172.21.15.78:0/2491165043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c4630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 >> 172.21.15.78:0/2491165043 conn(0x55dc2c4575f0 msgr2=0x55dc2c45d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3520ff9640 0 client.15708 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 shutdown_connections 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.536+0000 7f3541bca740 1 -- 172.21.15.78:0/2491165043 wait complete. 2024-09-16T01:13:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.537+0000 7f3541bca740 20 client.15708 trim_cache size 0 max 16384 2024-09-16T01:13:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 Processor -- start 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 -- start start 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c4579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b1920 con 0x55dc2c466240 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35381af1f0 con 0x55dc2c45b980 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f3541bca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35381ad040 con 0x55dc2c45b060 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34976/0 (socket says 172.21.15.78:34976) 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 -- 172.21.15.78:0/1337751259 learned_addr learned my addr 172.21.15.78:0/1337751259 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c4579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 -- 172.21.15.78:0/1337751259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 msgr2=0x55dc2c45dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 -- 172.21.15.78:0/1337751259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 msgr2=0x55dc2c4579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c4579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 -- 172.21.15.78:0/1337751259 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2c465020 con 0x55dc2c466240 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c4579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.554+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3530012650 tx=0x7f353001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54807891aad6abd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3522ffd640 1 -- 172.21.15.78:0/1337751259 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530012680 con 0x55dc2c466240 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3522ffd640 1 -- 172.21.15.78:0/1337751259 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530004510 con 0x55dc2c466240 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3522ffd640 1 -- 172.21.15.78:0/1337751259 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530020b70 con 0x55dc2c466240 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 -- 172.21.15.78:0/1337751259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 msgr2=0x55dc2c456640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3530012650 tx=0x7f353001f000 comp rx=0 tx=0).stop 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 -- 172.21.15.78:0/1337751259 shutdown_connections 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c466240 0x55dc2c456640 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c45b980 0x55dc2c4579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 --2- 172.21.15.78:0/1337751259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b060 0x55dc2c45dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 -- 172.21.15.78:0/1337751259 >> 172.21.15.78:0/1337751259 conn(0x55dc2c4575f0 msgr2=0x55dc2c455f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 -- 172.21.15.78:0/1337751259 shutdown_connections 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.555+0000 7f3541bca740 1 -- 172.21.15.78:0/1337751259 wait complete. 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 -- start start 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 Processor -- start 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 0x55dc2c46ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c46a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35381ad6b0 con 0x55dc2c45b060 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35381aeb20 con 0x55dc2c466240 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35381add80 con 0x55dc2c45b980 2024-09-16T01:13:17.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f353d58d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f3541bca740 10 client.0 _ll_register_callbacks cb 0x7f3528004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f353d58d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34988/0 (socket says 172.21.15.78:34988) 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f353d58d640 1 -- 172.21.15.78:0/1159371469 learned_addr learned my addr 172.21.15.78:0/1159371469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f353dd8e640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c46a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.556+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 0x55dc2c46ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 -- 172.21.15.78:0/1159371469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 msgr2=0x55dc2c46ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 0x55dc2c46ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 -- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 msgr2=0x55dc2c46a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c46a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 -- 172.21.15.78:0/1159371469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2c46b170 con 0x55dc2c45b060 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 0x55dc2c46ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f353d58d640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3534097da0 tx=0x7f3534017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced766368394d3e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:17.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353401e040 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f35340cf0b0 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3530004510 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3534010ca0 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 10 client.15714 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55dc2c4e1170 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530004510 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f35340b4200 con 0x55dc2c45b060 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 10 client.15714 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 10 client.15714 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 20 client.15714 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c4e1890 0x55dc2c4e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 4 client.15714 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d58bb50b", "release": true, "await": true}] 2024-09-16T01:13:17.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.557+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d58bb50b", "release": true, "await": true}) -- 0x55dc2c452950 con 0x55dc2c4e1890 2024-09-16T01:13:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.558+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c4e1890 0x55dc2c4e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.558+0000 7f3520ff9640 10 client.15714 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:17.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.558+0000 7f353cd8c640 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c4e1890 0x55dc2c4e3cb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ef4f7160d46583 server_cookie=3420f43beea464 in_seq=0 out_seq=0 2024-09-16T01:13:17.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3520ff9640 1 -- 172.21.15.78:0/1159371469 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55dc2c452950 con 0x55dc2c4e1890 2024-09-16T01:13:17.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3520ff9640 10 client.15714 handle_command_reply: tid=0 2024-09-16T01:13:17.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3541bca740 1 client.15714 shutdown 2024-09-16T01:13:17.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 msgr2=0x55dc2c46e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3534097da0 tx=0x7f3534017730 comp rx=0 tx=0).stop 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 shutdown_connections 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.604+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dc2c4e1890 0x55dc2c4e3cb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dc2c466240 0x55dc2c46a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc2c45b980 0x55dc2c46ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 --2- 172.21.15.78:0/1159371469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dc2c45b060 0x55dc2c46e530 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 >> 172.21.15.78:0/1159371469 conn(0x55dc2c4575f0 msgr2=0x55dc2c4638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3520ff9640 0 client.15714 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:17.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 shutdown_connections 2024-09-16T01:13:17.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.605+0000 7f3541bca740 1 -- 172.21.15.78:0/1159371469 wait complete. 2024-09-16T01:13:17.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.606+0000 7f3541bca740 20 client.15714 trim_cache size 0 max 16384 2024-09-16T01:13:17.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 30, 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 8, 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "d58bb50b": { 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 8, 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:17.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 139.6, 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:17.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:17.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:17.611 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:13:17.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.607+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 msgr2=0x7f35240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.607+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 0x7f35240a5870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3530005fd0 tx=0x7f3530009de0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.607+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 msgr2=0x7f3538198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.607+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f353400b4a0 tx=0x7f353400b960 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 shutdown_connections 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f35240a33b0 0x7f35240a5870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3538102ef0 0x7f353819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3538102570 0x7f3538198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 --2- 172.21.15.78:0/1053416728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3538100e20 0x7f3538198520 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 >> 172.21.15.78:0/1053416728 conn(0x7f35380fc7d0 msgr2=0x7f3538069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 shutdown_connections 2024-09-16T01:13:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:17.608+0000 7f353f7f1640 1 -- 172.21.15.78:0/1053416728 wait complete. 2024-09-16T01:13:18.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:17 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:18.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:17 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:18.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:17 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:13:18.452 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd58bb50b', total seconds elapsed: 23.1 2024-09-16T01:13:18.452 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-16T01:13:18.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:13:18.802 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:19.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:18 smithi078 ceph-mon[30937]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 6.7 MiB/s wr, 211 op/s 2024-09-16T01:13:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:18 smithi193 ceph-mon[35701]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 6.7 MiB/s wr, 211 op/s 2024-09-16T01:13:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:18 smithi148 ceph-mon[35030]: pgmap v78: 129 pgs: 129 active+clean; 1.4 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 6.7 MiB/s wr, 211 op/s 2024-09-16T01:13:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.431+0000 7f4e45c90640 1 -- 172.21.15.78:0/953589894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010ef30 msgr2=0x7f4e4010d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.431+0000 7f4e45c90640 1 --2- 172.21.15.78:0/953589894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010ef30 0x7f4e4010d2a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4e30007510 tx=0x7f4e3002b0c0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.432+0000 7f4e45c90640 1 -- 172.21.15.78:0/953589894 shutdown_connections 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.432+0000 7f4e45c90640 1 --2- 172.21.15.78:0/953589894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 0x7f4e40115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.432+0000 7f4e45c90640 1 --2- 172.21.15.78:0/953589894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010ef30 0x7f4e4010d2a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.432+0000 7f4e45c90640 1 --2- 172.21.15.78:0/953589894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010e580 0x7f4e4010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.432+0000 7f4e45c90640 1 -- 172.21.15.78:0/953589894 >> 172.21.15.78:0/953589894 conn(0x7f4e4006e140 msgr2=0x7f4e4006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:19.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.434+0000 7f4e45c90640 1 -- 172.21.15.78:0/953589894 shutdown_connections 2024-09-16T01:13:19.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.435+0000 7f4e45c90640 1 -- 172.21.15.78:0/953589894 wait complete. 2024-09-16T01:13:19.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.436+0000 7f4e45c90640 1 Processor -- start 2024-09-16T01:13:19.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.437+0000 7f4e45c90640 1 -- start start 2024-09-16T01:13:19.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.437+0000 7f4e45c90640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 0x7f4e401a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.437+0000 7f4e45c90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 0x7f4e401a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.437+0000 7f4e45c90640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 0x7f4e4019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.437+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 0x7f4e401a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.438+0000 7f4e3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 0x7f4e401a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35008/0 (socket says 172.21.15.78:35008) 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.438+0000 7f4e3effd640 1 -- 172.21.15.78:0/2786037238 learned_addr learned my addr 172.21.15.78:0/2786037238 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:19.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.438+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4e40118570 con 0x7f4e4010e580 2024-09-16T01:13:19.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.438+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4e401183f0 con 0x7f4e4010d9b0 2024-09-16T01:13:19.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.438+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e401186f0 con 0x7f4e4010ef30 2024-09-16T01:13:19.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.439+0000 7f4e3ffff640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 0x7f4e4019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:19.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3f7fe640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 0x7f4e401a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:19.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3ffff640 1 -- 172.21.15.78:0/2786037238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 msgr2=0x7f4e401a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:19.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3ffff640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 0x7f4e401a4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3ffff640 1 -- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 msgr2=0x7f4e401a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:19.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3ffff640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 0x7f4e401a4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.440+0000 7f4e3ffff640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4019f030 con 0x7f4e4010ef30 2024-09-16T01:13:19.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.441+0000 7f4e3ffff640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 0x7f4e4019e900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400ab20 tx=0x7f4e340071c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:19.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.443+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34007c00 con 0x7f4e4010ef30 2024-09-16T01:13:19.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.443+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e34007da0 con 0x7f4e4010ef30 2024-09-16T01:13:19.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.443+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34018670 con 0x7f4e4010ef30 2024-09-16T01:13:19.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.444+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4019f320 con 0x7f4e4010ef30 2024-09-16T01:13:19.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.444+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e401ab610 con 0x7f4e4010ef30 2024-09-16T01:13:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.448+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e40073f10 con 0x7f4e4010ef30 2024-09-16T01:13:19.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.451+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f4e34018810 con 0x7f4e4010ef30 2024-09-16T01:13:19.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.452+0000 7f4e3cff9640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 0x7f4e280a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:19.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.452+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f4e340d37f0 con 0x7f4e4010ef30 2024-09-16T01:13:19.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.454+0000 7f4e3f7fe640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 0x7f4e280a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:19.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.455+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4e3409fe60 con 0x7f4e4010ef30 2024-09-16T01:13:19.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.455+0000 7f4e3f7fe640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 0x7f4e280a5a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4e4019fa40 tx=0x7f4e38009500 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.801+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e4010e960 con 0x7f4e4010ef30 2024-09-16T01:13:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.803+0000 7f4e3cff9640 1 -- 172.21.15.78:0/2786037238 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11919 (secure 0 0 0) 0x7f4e3409f760 con 0x7f4e4010ef30 2024-09-16T01:13:19.806 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:13:19.807 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":30,"btime":"2024-09-16T01:13:16:888990+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:13:16.696410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:13:19.807 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 30 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 msgr2=0x7f4e280a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 0x7f4e280a5a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4e4019fa40 tx=0x7f4e38009500 comp rx=0 tx=0).stop 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 msgr2=0x7f4e4019e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 0x7f4e4019e900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400ab20 tx=0x7f4e340071c0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 shutdown_connections 2024-09-16T01:13:19.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4e280a3580 0x7f4e280a5a40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e4010ef30 0x7f4e4019e900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4010e580 0x7f4e401a4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 --2- 172.21.15.78:0/2786037238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e4010d9b0 0x7f4e401a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:19.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.808+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 >> 172.21.15.78:0/2786037238 conn(0x7f4e4006e140 msgr2=0x7f4e4010b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:19.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.812+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 shutdown_connections 2024-09-16T01:13:19.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:19.815+0000 7f4e45c90640 1 -- 172.21.15.78:0/2786037238 wait complete. 2024-09-16T01:13:20.174 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:19 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2786037238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:20.197 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:19 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2786037238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:20.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:19 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2786037238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:20.850 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-16T01:13:20.850 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-16T01:13:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 125.0 s, 839.2 tps, lat 495.987 ms stddev 2713.053 2024-09-16T01:13:21.239 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:21.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:20 smithi078 ceph-mon[30937]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 657 op/s 2024-09-16T01:13:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:20 smithi193 ceph-mon[35701]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 657 op/s 2024-09-16T01:13:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:20 smithi148 ceph-mon[35030]: pgmap v79: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 657 op/s 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 -- 172.21.15.78:0/4072833658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd0114ab0 msgr2=0x7f2fd0116ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/4072833658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd0114ab0 0x7f2fd0116ea0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009ca0 tx=0x7f2fc802a950 comp rx=0 tx=0).stop 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 -- 172.21.15.78:0/4072833658 shutdown_connections 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/4072833658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd0114ab0 0x7f2fd0116ea0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/4072833658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd0114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/4072833658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd01091f0 0x7f2fd01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 -- 172.21.15.78:0/4072833658 >> 172.21.15.78:0/4072833658 conn(0x7f2fd006e140 msgr2=0x7f2fd006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 -- 172.21.15.78:0/4072833658 shutdown_connections 2024-09-16T01:13:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.839+0000 7f2fd57e1640 1 -- 172.21.15.78:0/4072833658 wait complete. 2024-09-16T01:13:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 Processor -- start 2024-09-16T01:13:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 -- start start 2024-09-16T01:13:21.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd01091f0 0x7f2fd01a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd0114ab0 0x7f2fd01a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd011a790 con 0x7f2fd01091f0 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd011a610 con 0x7f2fd0114ab0 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.840+0000 7f2fd57e1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd011a910 con 0x7f2fd0109b10 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd01091f0 0x7f2fd01a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35242/0 (socket says 172.21.15.78:35242) 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 -- 172.21.15.78:0/762406506 learned_addr learned my addr 172.21.15.78:0/762406506 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd0114ab0 0x7f2fd01a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 -- 172.21.15.78:0/762406506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd0114ab0 msgr2=0x7f2fd01a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd0114ab0 0x7f2fd01a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 -- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd01091f0 msgr2=0x7f2fd01a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd01091f0 0x7f2fd01a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.841+0000 7f2fce7fc640 1 -- 172.21.15.78:0/762406506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd01a5cc0 con 0x7f2fd0109b10 2024-09-16T01:13:21.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.842+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400e9f0 tx=0x7f2fc400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.843+0000 7f2fb7fff640 1 -- 172.21.15.78:0/762406506 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400cd80 con 0x7f2fd0109b10 2024-09-16T01:13:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.843+0000 7f2fb7fff640 1 -- 172.21.15.78:0/762406506 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc4005500 con 0x7f2fd0109b10 2024-09-16T01:13:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.843+0000 7f2fb7fff640 1 -- 172.21.15.78:0/762406506 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4010690 con 0x7f2fd0109b10 2024-09-16T01:13:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.844+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd01a5fb0 con 0x7f2fd0109b10 2024-09-16T01:13:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.844+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd010a300 con 0x7f2fd0109b10 2024-09-16T01:13:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.847+0000 7f2fb7fff640 1 -- 172.21.15.78:0/762406506 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f2fc40046c0 con 0x7f2fd0109b10 2024-09-16T01:13:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.847+0000 7f2fd7bba740 1 Processor -- start 2024-09-16T01:13:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 -- start start 2024-09-16T01:13:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 0x55ce6fe51c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fb7fff640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 0x7f2fb80a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fdb6160 con 0x55ce6fe51850 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fd5d720 con 0x55ce6fe52170 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fdb2120 con 0x55ce6fe5ca30 2024-09-16T01:13:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.848+0000 7f2fb7fff640 1 -- 172.21.15.78:0/762406506 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2fc40d0440 con 0x7f2fd0109b10 2024-09-16T01:13:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.849+0000 7f2fceffd640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 0x7f2fb80a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fcf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 0x55ce6fe51c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:57814/0 (socket says 172.21.15.78:57814) 2024-09-16T01:13:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 -- 172.21.15.78:0/1176158266 learned_addr learned my addr 172.21.15.78:0/1176158266 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 -- 172.21.15.78:0/1176158266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5ca30 msgr2=0x55ce6fe5ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 -- 172.21.15.78:0/1176158266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 msgr2=0x55ce6fe51c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 0x55ce6fe51c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 -- 172.21.15.78:0/1176158266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce6fb27ba0 con 0x55ce6fe52170 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 0x55ce6fe51c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fceffd640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 0x7f2fb80a59c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009bd0 tx=0x7f2fc0005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.850+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40bafd0 tx=0x7f2fc409bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec09fd20576e9126 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.851+0000 7f2fb57fa640 1 -- 172.21.15.78:0/1176158266 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40a0740 con 0x55ce6fe52170 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.851+0000 7f2fb57fa640 1 -- 172.21.15.78:0/1176158266 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc40a5000 con 0x55ce6fe52170 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.851+0000 7f2fb57fa640 1 -- 172.21.15.78:0/1176158266 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4098a00 con 0x55ce6fe52170 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- 172.21.15.78:0/1176158266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 msgr2=0x55ce6fe5c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40bafd0 tx=0x7f2fc409bdb0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- 172.21.15.78:0/1176158266 shutdown_connections 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe5c4f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/1176158266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe51850 0x55ce6fe51c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- 172.21.15.78:0/1176158266 >> 172.21.15.78:0/1176158266 conn(0x55ce6fe4dde0 msgr2=0x55ce6fe4e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- 172.21.15.78:0/1176158266 shutdown_connections 2024-09-16T01:13:21.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- 172.21.15.78:0/1176158266 wait complete. 2024-09-16T01:13:21.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 -- start start 2024-09-16T01:13:21.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.852+0000 7f2fd7bba740 1 Processor -- start 2024-09-16T01:13:21.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 0x55ce6fe599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 0x55ce6fe59f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fe5f600 con 0x55ce6fe5ca30 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fd3e900 con 0x55ce6fe51850 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce6fe5f780 con 0x55ce6fe52170 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fd7bba740 10 client.0 _ll_register_callbacks cb 0x55ce6fd3c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fcf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fcf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35042/0 (socket says 172.21.15.78:35042) 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fcf7fe640 1 -- 172.21.15.78:0/2098581956 learned_addr learned my addr 172.21.15.78:0/2098581956 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fceffd640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 0x55ce6fe599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.853+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 0x55ce6fe59f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 -- 172.21.15.78:0/2098581956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 msgr2=0x55ce6fe59f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 0x55ce6fe59f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 -- 172.21.15.78:0/2098581956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 msgr2=0x55ce6fe5a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 -- 172.21.15.78:0/2098581956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce6fd7ec50 con 0x55ce6fe51850 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 0x55ce6fe59f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fceffd640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 0x55ce6fe599e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009fd0 tx=0x7f2fc0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32f2c1c57354cb26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc001ca20 con 0x55ce6fe51850 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2fc001cbc0 con 0x55ce6fe51850 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fc40a5000 con 0x55ce6fe51850 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fc0031050 con 0x55ce6fe51850 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fd7bba740 10 client.25184 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.854+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce6fe6b340 con 0x55ce6fe51850 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.855+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc40a5000 con 0x55ce6fe51850 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.855+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f2fc001a390 con 0x55ce6fe51850 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 10 client.25184 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 10 client.25184 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 20 client.25184 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fe6c800 0x55ce6fe6ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 4 client.25184 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce6fdc3890 con 0x55ce6fe6c800 2024-09-16T01:13:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.856+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fe6c800 0x55ce6fe6ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.857+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fe6c800 0x55ce6fe6ebf0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=731150161efe26a2 server_cookie=714318c213122a80 in_seq=0 out_seq=0 2024-09-16T01:13:21.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.857+0000 7f2fb2ff5640 10 client.25184 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/2098581956 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ce6fdc3890 con 0x55ce6fe6c800 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fb2ff5640 10 client.25184 handle_command_reply: tid=0 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 client.25184 shutdown 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 msgr2=0x55ce6fe599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 0x55ce6fe599e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009fd0 tx=0x7f2fc0004730 comp rx=0 tx=0).stop 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 shutdown_connections 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fe6c800 0x55ce6fe6ebf0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe5a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe52170 0x55ce6fe59f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/2098581956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe51850 0x55ce6fe599e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 >> 172.21.15.78:0/2098581956 conn(0x55ce6fe4dde0 msgr2=0x55ce6fe53b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.858+0000 7f2fb2ff5640 0 client.25184 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.859+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 shutdown_connections 2024-09-16T01:13:21.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.859+0000 7f2fd7bba740 1 -- 172.21.15.78:0/2098581956 wait complete. 2024-09-16T01:13:21.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.859+0000 7f2fd7bba740 20 client.25184 trim_cache size 0 max 16384 2024-09-16T01:13:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.883+0000 7f2fd7bba740 1 Processor -- start 2024-09-16T01:13:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.883+0000 7f2fd7bba740 1 -- start start 2024-09-16T01:13:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.883+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe592c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.883+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe74960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b7f10 con 0x55ce6fe5ca30 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b57e0 con 0x55ce6fe52170 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b3630 con 0x55ce6fe5b950 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35058/0 (socket says 172.21.15.78:35058) 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 -- 172.21.15.78:0/970181147 learned_addr learned my addr 172.21.15.78:0/970181147 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fceffd640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe74960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 -- 172.21.15.78:0/970181147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 msgr2=0x55ce6fe74960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe74960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 -- 172.21.15.78:0/970181147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 msgr2=0x55ce6fe592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 -- 172.21.15.78:0/970181147 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce6fe493f0 con 0x55ce6fe5ca30 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe74960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fce7fc640 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2fc409c320 tx=0x7f2fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b80632205d96877 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.884+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/970181147 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40ba070 con 0x55ce6fe5ca30 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.885+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/970181147 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc40b0050 con 0x55ce6fe5ca30 2024-09-16T01:13:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.885+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/970181147 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40ae070 con 0x55ce6fe5ca30 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.885+0000 7f2fd7bba740 1 -- 172.21.15.78:0/970181147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 msgr2=0x55ce6fe74420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.885+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2fc409c320 tx=0x7f2fc4004a40 comp rx=0 tx=0).stop 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 -- 172.21.15.78:0/970181147 shutdown_connections 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe74960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe74420 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/970181147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe52170 0x55ce6fe592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 -- 172.21.15.78:0/970181147 >> 172.21.15.78:0/970181147 conn(0x55ce6fe4dde0 msgr2=0x55ce6fe5b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 -- 172.21.15.78:0/970181147 shutdown_connections 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 -- 172.21.15.78:0/970181147 wait complete. 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 -- start start 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.886+0000 7f2fd7bba740 1 Processor -- start 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 0x55ce6fe64da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe61310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b3ca0 con 0x55ce6fe52170 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b5110 con 0x55ce6fe5ca30 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd01b4370 con 0x55ce6fe5b950 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fd7bba740 10 client.0 _ll_register_callbacks cb 0x7f2fc0009390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 0x55ce6fe64da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 0x55ce6fe64da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35066/0 (socket says 172.21.15.78:35066) 2024-09-16T01:13:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fceffd640 1 -- 172.21.15.78:0/3506518604 learned_addr learned my addr 172.21.15.78:0/3506518604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.887+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe61310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 -- 172.21.15.78:0/3506518604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 msgr2=0x55ce6fe652e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:13:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe652e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 -- 172.21.15.78:0/3506518604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 msgr2=0x55ce6fe64da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 0x55ce6fe64da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 -- 172.21.15.78:0/3506518604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce6fe61b40 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fcf7fe640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe61310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8005cc0 tx=0x7f2fc802ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6cb1ff2c8146487 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8034040 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f2fc802f5a0 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.888+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fc40b0050 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fc805b050 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 10 client.25190 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce6fed7fe0 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc8052070 con 0x55ce6fe5ca30 2024-09-16T01:13:21.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f2fc8004660 con 0x55ce6fe5ca30 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 10 client.25190 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 10 client.25190 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 20 client.25190 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fed8880 0x55ce6fedaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 4 client.25190 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x7f2fc40b0050 con 0x55ce6fed8880 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.889+0000 7f2fceffd640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fed8880 0x55ce6fedaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.890+0000 7f2fb2ff5640 10 client.25190 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:21.890+0000 7f2fceffd640 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fed8880 0x55ce6fedaca0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4a94ac83aa261 server_cookie=90905dffa9d74a2f in_seq=0 out_seq=0 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.191+0000 7f2fb2ff5640 1 -- 172.21.15.78:0/3506518604 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f2fc40b0050 con 0x55ce6fed8880 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.191+0000 7f2fb2ff5640 10 client.25190 handle_command_reply: tid=0 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 client.25190 shutdown 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 msgr2=0x55ce6fe61310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe61310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8005cc0 tx=0x7f2fc802ff90 comp rx=0 tx=0).stop 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 shutdown_connections 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ce6fed8880 0x55ce6fedaca0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ce6fe5ca30 0x55ce6fe61310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fb2ff5640 0 client.25190 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce6fe5b950 0x55ce6fe652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 --2- 172.21.15.78:0/3506518604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ce6fe52170 0x55ce6fe64da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 >> 172.21.15.78:0/3506518604 conn(0x55ce6fe4dde0 msgr2=0x55ce6fe599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 shutdown_connections 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 1 -- 172.21.15.78:0/3506518604 wait complete. 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd7bba740 20 client.25190 trim_cache size 0 max 16384 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:13:22.195 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 30, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 10, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "20ef107a": { 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 10, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:13:22.196 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 139.6, 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:22.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:22.198 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 msgr2=0x7f2fb80a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 0x7f2fb80a59c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009bd0 tx=0x7f2fc0005f90 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 msgr2=0x7f2fd01a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.192+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400e9f0 tx=0x7f2fc400edf0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 shutdown_connections 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2fb80a3500 0x7f2fb80a59c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fd0114ab0 0x7f2fd01a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fd0109b10 0x7f2fd01a1250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 --2- 172.21.15.78:0/762406506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fd01091f0 0x7f2fd01a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:22.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 >> 172.21.15.78:0/762406506 conn(0x7f2fd006e140 msgr2=0x7f2fd0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:22.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 shutdown_connections 2024-09-16T01:13:22.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:22.193+0000 7f2fd57e1640 1 -- 172.21.15.78:0/762406506 wait complete. 2024-09-16T01:13:23.251 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:22 smithi078 ceph-mon[30937]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 MiB/s wr, 654 op/s 2024-09-16T01:13:23.285 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '20ef107a', quiesce took 4.8 seconds. Will release after: 14.75 2024-09-16T01:13:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:22 smithi193 ceph-mon[35701]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 MiB/s wr, 654 op/s 2024-09-16T01:13:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:22 smithi148 ceph-mon[35030]: pgmap v80: 129 pgs: 129 active+clean; 1.5 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 MiB/s wr, 654 op/s 2024-09-16T01:13:25.188 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:24 smithi148 ceph-mon[35030]: pgmap v81: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 718 op/s 2024-09-16T01:13:25.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:24 smithi078 ceph-mon[30937]: pgmap v81: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 718 op/s 2024-09-16T01:13:25.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:24 smithi193 ceph-mon[35701]: pgmap v81: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 MiB/s wr, 718 op/s 2024-09-16T01:13:25.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 130.0 s, 489.9 tps, lat 49.011 ms stddev 60.268 2024-09-16T01:13:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:26 smithi078 ceph-mon[30937]: pgmap v82: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T01:13:27.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:26 smithi193 ceph-mon[35701]: pgmap v82: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T01:13:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:26 smithi148 ceph-mon[35030]: pgmap v82: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 -- 172.21.15.78:0/2596201789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 msgr2=0x7fbef8106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 --2- 172.21.15.78:0/2596201789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef8106da0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009cd0 tx=0x7fbee802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 -- 172.21.15.78:0/2596201789 shutdown_connections 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 --2- 172.21.15.78:0/2596201789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 0x7fbef810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 --2- 172.21.15.78:0/2596201789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 0x7fbef8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 --2- 172.21.15.78:0/2596201789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef8106da0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 -- 172.21.15.78:0/2596201789 >> 172.21.15.78:0/2596201789 conn(0x7fbef8077990 msgr2=0x7fbef80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 -- 172.21.15.78:0/2596201789 shutdown_connections 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.023+0000 7fbefedc0640 1 -- 172.21.15.78:0/2596201789 wait complete. 2024-09-16T01:13:29.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 Processor -- start 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 -- start start 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 0x7fbef8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 0x7fbef81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbef8110e00 con 0x7fbef81069c0 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbef8110c80 con 0x7fbef8101350 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefedc0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbef8110f80 con 0x7fbef81009b0 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefd35d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefd35d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51246/0 (socket says 172.21.15.78:51246) 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbefd35d640 1 -- 172.21.15.78:0/1770658228 learned_addr learned my addr 172.21.15.78:0/1770658228 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.024+0000 7fbef7fff640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 0x7fbef81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 0x7fbef8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 -- 172.21.15.78:0/1770658228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 msgr2=0x7fbef8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 0x7fbef8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 -- 172.21.15.78:0/1770658228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 msgr2=0x7fbef81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 0x7fbef81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 -- 172.21.15.78:0/1770658228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef819d380 con 0x7fbef81069c0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefd35d640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbef000d750 tx=0x7fbef000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbef5ffb640 1 -- 172.21.15.78:0/1770658228 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef0017070 con 0x7fbef81069c0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbef5ffb640 1 -- 172.21.15.78:0/1770658228 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef0005500 con 0x7fbef81069c0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef819d670 con 0x7fbef81069c0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbef5ffb640 1 -- 172.21.15.78:0/1770658228 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef0004d90 con 0x7fbef81069c0 2024-09-16T01:13:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.025+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef81a4f00 con 0x7fbef81069c0 2024-09-16T01:13:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.027+0000 7fbef5ffb640 1 -- 172.21.15.78:0/1770658228 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fbef0004f30 con 0x7fbef81069c0 2024-09-16T01:13:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 Processor -- start 2024-09-16T01:13:29.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 -- start start 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef5ffb640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 0x7fbee40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b8af0 0x559d287b8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef5ffb640 1 -- 172.21.15.78:0/1770658228 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbef00cfa20 con 0x7fbef81069c0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef7fff640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 0x7fbee40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559d2871d2f0 con 0x559d287b9410 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559d286c5270 con 0x559d287c3cb0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559d287192b0 con 0x559d287b8af0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51248/0 (socket says 172.21.15.78:51248) 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbefd35d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.028+0000 7fbef7fff640 1 -- 172.21.15.78:0/2047440706 learned_addr learned my addr 172.21.15.78:0/2047440706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b8af0 0x559d287b8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 -- 172.21.15.78:0/2047440706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b8af0 msgr2=0x559d287b8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b8af0 0x559d287b8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 -- 172.21.15.78:0/2047440706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 msgr2=0x559d287c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 -- 172.21.15.78:0/2047440706 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d286c4340 con 0x559d287c3cb0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbef7fff640 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbef7fff640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 0x7fbee40a5920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbee0004650 tx=0x7fbee00045a0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbefd35d640 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287c60a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbef00ccde0 tx=0x7fbef00056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a60902ad9afc4627 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbeeeffd640 1 -- 172.21.15.78:0/2047440706 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef0020070 con 0x559d287c3cb0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbeeeffd640 1 -- 172.21.15.78:0/2047440706 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef009fd20 con 0x559d287c3cb0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.029+0000 7fbeeeffd640 1 -- 172.21.15.78:0/2047440706 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef00cd060 con 0x559d287c3cb0 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2047440706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 msgr2=0x559d287c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287c60a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbef00ccde0 tx=0x7fbef00056b0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2047440706 shutdown_connections 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287c60a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d287c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2047440706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b8af0 0x559d287b8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2047440706 >> 172.21.15.78:0/2047440706 conn(0x559d287b5060 msgr2=0x559d287b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2047440706 shutdown_connections 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2047440706 wait complete. 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 -- start start 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.030+0000 7fbf01c2e740 1 Processor -- start 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 0x559d287c1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559d287c6880 con 0x559d287b8af0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559d286a6290 con 0x559d287b9410 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559d287c6a00 con 0x559d287c3cb0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbf01c2e740 10 client.0 _ll_register_callbacks cb 0x559d286d7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51252/0 (socket says 172.21.15.78:51252) 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 -- 172.21.15.78:0/2191931747 learned_addr learned my addr 172.21.15.78:0/2191931747 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbef7fff640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 0x559d287c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefd35d640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 -- 172.21.15.78:0/2191931747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 msgr2=0x559d287c1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 -- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 msgr2=0x559d287c1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 0x559d287c1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 -- 172.21.15.78:0/2191931747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d28430210 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbef7fff640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 0x559d287c1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbee8005cc0 tx=0x7fbee8031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e5c4d1c6260b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.031+0000 7fbed7fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8048060 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbee8031ce0 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbef009fd20 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbee804f050 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 10 client.15747 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d284fc620 con 0x559d287b8af0 2024-09-16T01:13:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef009fd20 con 0x559d287b8af0 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fbee8042070 con 0x559d287b8af0 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 10 client.15747 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 20 client.15747 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d287d3ca0 0x559d287d60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 4 client.15747 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.032+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d2872aa20 con 0x559d287d3ca0 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.033+0000 7fbef7fff640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d287d3ca0 0x559d287d60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.033+0000 7fbed7fff640 10 client.15747 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:29.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.033+0000 7fbef7fff640 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d287d3ca0 0x559d287d60d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9ef7727e9473ca server_cookie=caea6ca51d5f3753 in_seq=0 out_seq=0 2024-09-16T01:13:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbed7fff640 1 -- 172.21.15.78:0/2191931747 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559d2872aa20 con 0x559d287d3ca0 2024-09-16T01:13:29.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbed7fff640 10 client.15747 handle_command_reply: tid=0 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 client.15747 shutdown 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 msgr2=0x559d287c0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbee8005cc0 tx=0x7fbee8031af0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 shutdown_connections 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d287d3ca0 0x559d287d60d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b9410 0x559d287c1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/2191931747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c0bf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbed7fff640 0 client.15747 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 >> 172.21.15.78:0/2191931747 conn(0x559d287b5060 msgr2=0x559d287badc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 shutdown_connections 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 1 -- 172.21.15.78:0/2191931747 wait complete. 2024-09-16T01:13:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.034+0000 7fbf01c2e740 20 client.15747 trim_cache size 0 max 16384 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 Processor -- start 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 -- start start 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b9410 0x559d287bacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287db9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81b1730 con 0x559d287b8af0 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81af000 con 0x559d287c3cb0 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81ace50 con 0x559d287b9410 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51260/0 (socket says 172.21.15.78:51260) 2024-09-16T01:13:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 -- 172.21.15.78:0/3684758063 learned_addr learned my addr 172.21.15.78:0/3684758063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 -- 172.21.15.78:0/3684758063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b9410 msgr2=0x559d287bacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b9410 0x559d287bacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 -- 172.21.15.78:0/3684758063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 msgr2=0x559d287db9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287db9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.049+0000 7fbefcb5c640 1 -- 172.21.15.78:0/3684758063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d287bb940 con 0x559d287b8af0 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f80 tx=0x7fbee8005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bac2b9eda3def2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbeee7fc640 1 -- 172.21.15.78:0/3684758063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8048040 con 0x559d287b8af0 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbeee7fc640 1 -- 172.21.15.78:0/3684758063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbee8036040 con 0x559d287b8af0 2024-09-16T01:13:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbeee7fc640 1 -- 172.21.15.78:0/3684758063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8030dd0 con 0x559d287b8af0 2024-09-16T01:13:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 -- 172.21.15.78:0/3684758063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 msgr2=0x559d287c14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbee8000f80 tx=0x7fbee8005de0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 -- 172.21.15.78:0/3684758063 shutdown_connections 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287c3cb0 0x559d287db9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287b9410 0x559d287bacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/3684758063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b8af0 0x559d287c14a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 -- 172.21.15.78:0/3684758063 >> 172.21.15.78:0/3684758063 conn(0x559d287b5060 msgr2=0x559d287ba5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 -- 172.21.15.78:0/3684758063 shutdown_connections 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.050+0000 7fbf01c2e740 1 -- 172.21.15.78:0/3684758063 wait complete. 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 -- start start 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 Processor -- start 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b8af0 0x559d28836d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81ad4c0 con 0x559d287b9410 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81ae930 con 0x559d287b8af0 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbef81adb90 con 0x559d287c3cb0 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbf01c2e740 10 client.0 _ll_register_callbacks cb 0x7fbee8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbef7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbef7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51266/0 (socket says 172.21.15.78:51266) 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbef7fff640 1 -- 172.21.15.78:0/149855094 learned_addr learned my addr 172.21.15.78:0/149855094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.051+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b8af0 0x559d28836d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 -- 172.21.15.78:0/149855094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 msgr2=0x559d287c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 -- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b8af0 msgr2=0x559d28836d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b8af0 0x559d28836d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 -- 172.21.15.78:0/149855094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d287c79e0 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbef7fff640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbee0005fd0 tx=0x7fbee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166c688ef4d7befd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee002e020 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fbee002a550 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbee8036040 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbee0039050 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbf01c2e740 10 client.15753 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d2883e550 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbee8036040 con 0x559d287b9410 2024-09-16T01:13:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.052+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fbee001e040 con 0x559d287b9410 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 10 client.15753 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 20 client.15753 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d2883ed30 0x559d28841150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 4 client.15753 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d28724d70 con 0x559d2883ed30 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d2883ed30 0x559d28841150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbed7fff640 10 client.15753 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.053+0000 7fbefcb5c640 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d2883ed30 0x559d28841150 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9faa5c50d47e91bc server_cookie=f082ab706383e8dc in_seq=0 out_seq=0 2024-09-16T01:13:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbed7fff640 1 -- 172.21.15.78:0/149855094 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d28724d70 con 0x559d2883ed30 2024-09-16T01:13:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbed7fff640 10 client.15753 handle_command_reply: tid=0 2024-09-16T01:13:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 client.15753 shutdown 2024-09-16T01:13:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 msgr2=0x559d28837260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbee0005fd0 tx=0x7fbee00046c0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 shutdown_connections 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559d2883ed30 0x559d28841150 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d287c3cb0 0x559d287c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d287b9410 0x559d28837260 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbed7fff640 0 client.15753 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 --2- 172.21.15.78:0/149855094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559d287b8af0 0x559d28836d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 >> 172.21.15.78:0/149855094 conn(0x559d287b5060 msgr2=0x559d287b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 shutdown_connections 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 1 -- 172.21.15.78:0/149855094 wait complete. 2024-09-16T01:13:29.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.061+0000 7fbf01c2e740 20 client.15753 trim_cache size 0 max 16384 2024-09-16T01:13:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:13:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:13:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:13:29.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 msgr2=0x7fbee40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 0x7fbee40a5920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbee0004650 tx=0x7fbee00045a0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 msgr2=0x7fbef819cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbef000d750 tx=0x7fbef000dc10 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 shutdown_connections 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbee40a3460 0x7fbee40a5920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbef81069c0 0x7fbef819cc50 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbef8101350 0x7fbef81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 --2- 172.21.15.78:0/1770658228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbef81009b0 0x7fbef8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 >> 172.21.15.78:0/1770658228 conn(0x7fbef8077990 msgr2=0x7fbef8069b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:29.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 shutdown_connections 2024-09-16T01:13:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:29.062+0000 7fbefedc0640 1 -- 172.21.15.78:0/1770658228 wait complete. 2024-09-16T01:13:29.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:28 smithi078 ceph-mon[30937]: pgmap v83: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 729 op/s 2024-09-16T01:13:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:28 smithi193 ceph-mon[35701]: pgmap v83: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 729 op/s 2024-09-16T01:13:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:28 smithi148 ceph-mon[35030]: pgmap v83: 129 pgs: 129 active+clean; 1.5 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 729 op/s 2024-09-16T01:13:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 135.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:13:31.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:30 smithi078 ceph-mon[30937]: pgmap v84: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 812 op/s 2024-09-16T01:13:31.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:30 smithi193 ceph-mon[35701]: pgmap v84: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 812 op/s 2024-09-16T01:13:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:30 smithi148 ceph-mon[35030]: pgmap v84: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 812 op/s 2024-09-16T01:13:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:32 smithi078 ceph-mon[30937]: pgmap v85: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 366 op/s 2024-09-16T01:13:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:32 smithi193 ceph-mon[35701]: pgmap v85: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 366 op/s 2024-09-16T01:13:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:32 smithi148 ceph-mon[35030]: pgmap v85: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 4.3 MiB/s wr, 366 op/s 2024-09-16T01:13:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:34 smithi078 ceph-mon[30937]: pgmap v86: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.3 MiB/s wr, 369 op/s 2024-09-16T01:13:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:34 smithi193 ceph-mon[35701]: pgmap v86: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.3 MiB/s wr, 369 op/s 2024-09-16T01:13:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:34 smithi148 ceph-mon[35030]: pgmap v86: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.3 MiB/s wr, 369 op/s 2024-09-16T01:13:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 140.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:13:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:36 smithi078 ceph-mon[30937]: pgmap v87: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-09-16T01:13:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:36 smithi193 ceph-mon[35701]: pgmap v87: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-09-16T01:13:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:36 smithi148 ceph-mon[35030]: pgmap v87: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-09-16T01:13:38.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:13:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 -- 172.21.15.78:0/3586261571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900101420 msgr2=0x7f590010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 --2- 172.21.15.78:0/3586261571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900101420 0x7f590010d950 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009cd0 tx=0x7f58f802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 -- 172.21.15.78:0/3586261571 shutdown_connections 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 --2- 172.21.15.78:0/3586261571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900101420 0x7f590010d950 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 --2- 172.21.15.78:0/3586261571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900100a80 0x7f5900100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 --2- 172.21.15.78:0/3586261571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900106a80 0x7f5900106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 -- 172.21.15.78:0/3586261571 >> 172.21.15.78:0/3586261571 conn(0x7f59000fc7a0 msgr2=0x7f59000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 -- 172.21.15.78:0/3586261571 shutdown_connections 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.803+0000 7f59054e2640 1 -- 172.21.15.78:0/3586261571 wait complete. 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.804+0000 7f59054e2640 1 Processor -- start 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.804+0000 7f59054e2640 1 -- start start 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 0x7f59001997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900106a80 0x7f59001a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5900110ed0 con 0x7f5900100a80 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5900110d50 con 0x7f5900106a80 2024-09-16T01:13:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5900111050 con 0x7f5900101420 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58420/0 (socket says 172.21.15.78:58420) 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 -- 172.21.15.78:0/3214871503 learned_addr learned my addr 172.21.15.78:0/3214871503 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58fe7fc640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 0x7f59001997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58ff7fe640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900106a80 0x7f59001a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 -- 172.21.15.78:0/3214871503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 msgr2=0x7f59001997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 0x7f59001997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 -- 172.21.15.78:0/3214871503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900106a80 msgr2=0x7f59001a42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900106a80 0x7f59001a42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 -- 172.21.15.78:0/3214871503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59001a4820 con 0x7f5900100a80 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58fe7fc640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 0x7f59001997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:13:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58feffd640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f58f000d770 tx=0x7f58f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0017070 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59001a49f0 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0005500 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.805+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0004e00 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.806+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59001a4eb0 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.807+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5900100ee0 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.807+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f58f0004fa0 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.808+0000 7f58e7fff640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.808+0000 7f58fe7fc640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.808+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f58f00cfb80 con 0x7f5900100a80 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.809+0000 7f58fe7fc640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 0x7f58e80a5870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f590006b2e0 tx=0x7f58f400a5c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:38.812+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f58f009c1d0 con 0x7f5900100a80 2024-09-16T01:13:39.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.016+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59001a50e0 con 0x7f5900100a80 2024-09-16T01:13:39.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.017+0000 7f58e7fff640 1 -- 172.21.15.78:0/3214871503 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11919 (secure 0 0 0) 0x7f58f009bad0 con 0x7f5900100a80 2024-09-16T01:13:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:13:39.018 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":30,"btime":"2024-09-16T01:13:16:888990+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:13:16.696410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24455,24460,24485]},"id":1}]} 2024-09-16T01:13:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 30 2024-09-16T01:13:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 msgr2=0x7f58e80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 0x7f58e80a5870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f590006b2e0 tx=0x7f58f400a5c0 comp rx=0 tx=0).stop 2024-09-16T01:13:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 msgr2=0x7f5900199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f58f000d770 tx=0x7f58f000dc30 comp rx=0 tx=0).stop 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 shutdown_connections 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f58e80a33b0 0x7f58e80a5870 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5900106a80 0x7f59001a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5900101420 0x7f59001997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 --2- 172.21.15.78:0/3214871503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5900100a80 0x7f5900199260 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 >> 172.21.15.78:0/3214871503 conn(0x7f59000fc7a0 msgr2=0x7f59000fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 shutdown_connections 2024-09-16T01:13:39.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:39.019+0000 7f59054e2640 1 -- 172.21.15.78:0/3214871503 wait complete. 2024-09-16T01:13:39.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:38 smithi078 ceph-mon[30937]: pgmap v88: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 695 KiB/s wr, 88 op/s 2024-09-16T01:13:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:38 smithi193 ceph-mon[35701]: pgmap v88: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 695 KiB/s wr, 88 op/s 2024-09-16T01:13:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:38 smithi148 ceph-mon[35030]: pgmap v88: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 695 KiB/s wr, 88 op/s 2024-09-16T01:13:39.829 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 20ef107a --release --await' 2024-09-16T01:13:39.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 20ef107a --release --await 2024-09-16T01:13:40.078 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:13:40.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:39 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3214871503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:40.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:39 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3214871503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:40.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:39 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3214871503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:13:40.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 -- 172.21.15.78:0/2474719976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060101de0 msgr2=0x7f30601021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2474719976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060101de0 0x7f30601021c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3054009900 tx=0x7f305402a410 comp rx=0 tx=0).stop 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 -- 172.21.15.78:0/2474719976 shutdown_connections 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2474719976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060106f10 0x7f306010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2474719976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060102790 0x7f30601067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2474719976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060101de0 0x7f30601021c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 -- 172.21.15.78:0/2474719976 >> 172.21.15.78:0/2474719976 conn(0x7f30600fd8c0 msgr2=0x7f30600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 -- 172.21.15.78:0/2474719976 shutdown_connections 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.607+0000 7f3066d2a640 1 -- 172.21.15.78:0/2474719976 wait complete. 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 Processor -- start 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 -- start start 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060101de0 0x7f3060198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060102790 0x7f3060198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3060112010 con 0x7f3060106f10 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3060111e90 con 0x7f3060102790 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3066d2a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3060112190 con 0x7f3060101de0 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f30652c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f30652c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58428/0 (socket says 172.21.15.78:58428) 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f30652c7640 1 -- 172.21.15.78:0/2688459861 learned_addr learned my addr 172.21.15.78:0/2688459861 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f3064ac6640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060101de0 0x7f3060198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.608+0000 7f305ffff640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060102790 0x7f3060198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 -- 172.21.15.78:0/2688459861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060101de0 msgr2=0x7f3060198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060101de0 0x7f3060198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 -- 172.21.15.78:0/2688459861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060102790 msgr2=0x7f3060198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060102790 0x7f3060198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 -- 172.21.15.78:0/2688459861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306019d630 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f30652c7640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f305800b970 tx=0x7f305800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f305dffb640 1 -- 172.21.15.78:0/2688459861 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058019070 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f305dffb640 1 -- 172.21.15.78:0/2688459861 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305800cb40 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306019d920 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f305dffb640 1 -- 172.21.15.78:0/2688459861 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058007650 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.609+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30601a5200 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f305dffb640 1 -- 172.21.15.78:0/2688459861 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f305801c070 con 0x7f3060106f10 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 Processor -- start 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 -- start start 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb30440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56177ea95620 con 0x56177eb3b240 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56177ea3ce50 con 0x56177eb30060 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.612+0000 7f3069103740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56177ea915e0 con 0x56177eb30980 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305dffb640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 0x7f30500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58602/0 (socket says 172.21.15.78:58602) 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 -- 172.21.15.78:0/541846159 learned_addr learned my addr 172.21.15.78:0/541846159 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58442/0 (socket says 172.21.15.78:58442) 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f30652c7640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 0x7f30500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305dffb640 1 -- 172.21.15.78:0/2688459861 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f30580cf640 con 0x7f3060106f10 2024-09-16T01:13:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305ffff640 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb30440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 -- 172.21.15.78:0/541846159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 msgr2=0x56177eb30440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb30440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 -- 172.21.15.78:0/541846159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 msgr2=0x56177eb3d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 -- 172.21.15.78:0/541846159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177ea2dd40 con 0x56177eb30980 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f305ffff640 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f30652c7640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 0x7f30500a5870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30580ccbc0 tx=0x7f305806d420 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f3064ac6640 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3054005ab0 tx=0x7f3054031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b8d0933c2a89a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f304affd640 1 -- 172.21.15.78:0/541846159 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305403f070 con 0x56177eb30980 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f304affd640 1 -- 172.21.15.78:0/541846159 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3054031a20 con 0x56177eb30980 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.613+0000 7f304affd640 1 -- 172.21.15.78:0/541846159 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054035070 con 0x56177eb30980 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 -- 172.21.15.78:0/541846159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 msgr2=0x56177eb3ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3054005ab0 tx=0x7f3054031630 comp rx=0 tx=0).stop 2024-09-16T01:13:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 -- 172.21.15.78:0/541846159 shutdown_connections 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb3d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3ad00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 --2- 172.21.15.78:0/541846159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb30440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 -- 172.21.15.78:0/541846159 >> 172.21.15.78:0/541846159 conn(0x56177eb2c5f0 msgr2=0x56177eb2c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 -- 172.21.15.78:0/541846159 shutdown_connections 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.614+0000 7f3069103740 1 -- 172.21.15.78:0/541846159 wait complete. 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 -- start start 2024-09-16T01:13:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 Processor -- start 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 0x56177eb38b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56177eb3de10 con 0x56177eb30980 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56177ea1df10 con 0x56177eb3b240 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56177eb3df90 con 0x56177eb30060 2024-09-16T01:13:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3069103740 10 client.0 _ll_register_callbacks cb 0x56177ea41db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58448/0 (socket says 172.21.15.78:58448) 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 -- 172.21.15.78:0/3760004007 learned_addr learned my addr 172.21.15.78:0/3760004007 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f305ffff640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 0x56177eb38b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f305ffff640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 -- 172.21.15.78:0/3760004007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 msgr2=0x56177eb380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb380f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 -- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 msgr2=0x56177eb38b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 0x56177eb38b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.615+0000 7f3064ac6640 1 -- 172.21.15.78:0/3760004007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177e7a4580 con 0x56177eb30980 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f305ffff640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 0x56177eb38b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3064ac6640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3054009900 tx=0x7f3054048a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb2f91a75380615 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305404f070 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3054004020 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3054031a20 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f305405a050 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 10 client.15771 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56177e86e960 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3054031a20 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f30540039f0 con 0x56177eb30980 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 10 client.15771 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 10 client.15771 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 20 client.15771 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.616+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177eb4d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.617+0000 7f3069103740 4 client.15771 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.617+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56177eaa2b90 con 0x56177eb4ac10 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.617+0000 7f305ffff640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177eb4d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.617+0000 7f303ffff640 10 client.15771 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.617+0000 7f305ffff640 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177eb4d000 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bdcdda913473453 server_cookie=b10bbf33303f2089 in_seq=0 out_seq=0 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f303ffff640 1 -- 172.21.15.78:0/3760004007 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56177eaa2b90 con 0x56177eb4ac10 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f303ffff640 10 client.15771 handle_command_reply: tid=0 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 client.15771 shutdown 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 msgr2=0x56177eb38630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3054009900 tx=0x7f3054048a50 comp rx=0 tx=0).stop 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 shutdown_connections 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177eb4d000 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb3b240 0x56177eb38b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb30980 0x56177eb38630 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 --2- 172.21.15.78:0/3760004007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f303ffff640 0 client.15771 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:40.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 >> 172.21.15.78:0/3760004007 conn(0x56177eb2c5f0 msgr2=0x56177eb322d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 shutdown_connections 2024-09-16T01:13:40.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.618+0000 7f3069103740 1 -- 172.21.15.78:0/3760004007 wait complete. 2024-09-16T01:13:40.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.619+0000 7f3069103740 20 client.15771 trim_cache size 0 max 16384 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 Processor -- start 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 -- start start 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb52600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30980 0x56177eb52b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f30601af6c0 con 0x56177eb3b240 2024-09-16T01:13:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30601acf90 con 0x56177eb30980 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3069103740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30601aade0 con 0x56177eb30060 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58460/0 (socket says 172.21.15.78:58460) 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f305ffff640 1 -- 172.21.15.78:0/970638389 learned_addr learned my addr 172.21.15.78:0/970638389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.635+0000 7f3064ac6640 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30980 0x56177eb52b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb52600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 -- 172.21.15.78:0/970638389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 msgr2=0x56177eb52600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb52600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 -- 172.21.15.78:0/970638389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30980 msgr2=0x56177eb52b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30980 0x56177eb52b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 -- 172.21.15.78:0/970638389 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177eb2abb0 con 0x56177eb3b240 2024-09-16T01:13:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f305ffff640 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x56177eb2d970 tx=0x7f304c006da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb53ce47e8ee57a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f304a7fc640 1 -- 172.21.15.78:0/970638389 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c025020 con 0x56177eb3b240 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f304a7fc640 1 -- 172.21.15.78:0/970638389 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f304c017ba0 con 0x56177eb3b240 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.636+0000 7f304a7fc640 1 -- 172.21.15.78:0/970638389 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c018ae0 con 0x56177eb3b240 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- 172.21.15.78:0/970638389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 msgr2=0x56177eb35350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x56177eb2d970 tx=0x7f304c006da0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- 172.21.15.78:0/970638389 shutdown_connections 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb35350 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30980 0x56177eb52b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- 172.21.15.78:0/970638389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30060 0x56177eb52600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- 172.21.15.78:0/970638389 >> 172.21.15.78:0/970638389 conn(0x56177eb2c5f0 msgr2=0x56177eb39950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- 172.21.15.78:0/970638389 shutdown_connections 2024-09-16T01:13:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- 172.21.15.78:0/970638389 wait complete. 2024-09-16T01:13:40.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- start start 2024-09-16T01:13:40.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 Processor -- start 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb3f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f30601ab450 con 0x56177eb3b240 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30601ac8c0 con 0x56177eb30060 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.637+0000 7f3069103740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f30601abb20 con 0x56177eb30980 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f3069103740 10 client.0 _ll_register_callbacks cb 0x7f3054005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:40.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58472/0 (socket says 172.21.15.78:58472) 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 -- 172.21.15.78:0/2408576997 learned_addr learned my addr 172.21.15.78:0/2408576997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb3f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f3064ac6640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 -- 172.21.15.78:0/2408576997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 msgr2=0x56177eb3fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 -- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 msgr2=0x56177eb3f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb3f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 -- 172.21.15.78:0/2408576997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177eb43550 con 0x56177eb3b240 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f305ffff640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f304c014520 tx=0x7f304c021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80246ef92f53cdf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f303ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:13:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304c032060 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f304c0176c0 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f304c017ba0 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f304c039050 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f3069103740 10 client.15777 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.638+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56177ebb6070 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f304c017ba0 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f304c02a040 con 0x56177eb3b240 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 10 client.15777 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 10 client.15777 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 20 client.15777 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177ebb86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 4 client.15777 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "20ef107a", "release": true, "await": true}] 2024-09-16T01:13:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "20ef107a", "release": true, "await": true}) -- 0x56177ea9d0a0 con 0x56177eb4ac10 2024-09-16T01:13:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.639+0000 7f3064ac6640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177ebb86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.640+0000 7f303ffff640 10 client.15777 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.640+0000 7f3064ac6640 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177ebb86e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d778722aeb458ff server_cookie=61816cd2ea1ad89b in_seq=0 out_seq=0 2024-09-16T01:13:40.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f303ffff640 1 -- 172.21.15.78:0/2408576997 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56177ea9d0a0 con 0x56177eb4ac10 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f303ffff640 10 client.15777 handle_command_reply: tid=0 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 client.15777 shutdown 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 msgr2=0x56177eb42eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f304c014520 tx=0x7f304c021040 comp rx=0 tx=0).stop 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 shutdown_connections 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56177eb4ac10 0x56177ebb86e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f303ffff640 0 client.15777 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56177eb3b240 0x56177eb42eb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56177eb30980 0x56177eb3fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 --2- 172.21.15.78:0/2408576997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56177eb30060 0x56177eb3f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 >> 172.21.15.78:0/2408576997 conn(0x56177eb2c5f0 msgr2=0x56177eb2af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 shutdown_connections 2024-09-16T01:13:40.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 1 -- 172.21.15.78:0/2408576997 wait complete. 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.697+0000 7f3069103740 20 client.15777 trim_cache size 0 max 16384 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 30, 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 12, 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "20ef107a": { 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 12, 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:13:40.700 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 139.6, 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:13:40.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:40.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:40.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:40.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:40.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:13:40.702 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.698+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 msgr2=0x7f30500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.698+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 0x7f30500a5870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30580ccbc0 tx=0x7f305806d420 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.698+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 msgr2=0x7f306019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.698+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f305800b970 tx=0x7f305800be30 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 shutdown_connections 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f30500a33b0 0x7f30500a5870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3060106f10 0x7f306019cf50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3060102790 0x7f3060198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 --2- 172.21.15.78:0/2688459861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3060101de0 0x7f3060198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:40.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.699+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 >> 172.21.15.78:0/2688459861 conn(0x7f30600fd8c0 msgr2=0x7f30600fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:40.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.702+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 shutdown_connections 2024-09-16T01:13:40.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:13:40.703+0000 7f3066d2a640 1 -- 172.21.15.78:0/2688459861 wait complete. 2024-09-16T01:13:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 145.0 s, 97.7 tps, lat 3783.215 ms stddev 7491.831 2024-09-16T01:13:41.220 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:40 smithi078 ceph-mon[30937]: pgmap v89: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-09-16T01:13:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:40 smithi193 ceph-mon[35701]: pgmap v89: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-09-16T01:13:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:40 smithi148 ceph-mon[35030]: pgmap v89: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-09-16T01:13:41.699 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '20ef107a', total seconds elapsed: 23.2 2024-09-16T01:13:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.199+0000 7f57edf4b640 1 -- 172.21.15.78:0/2333218760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 msgr2=0x7f57e8100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.199+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2333218760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8100e50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f57e001acb0 tx=0x7f57e003b9b0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.205+0000 7f57edf4b640 1 -- 172.21.15.78:0/2333218760 shutdown_connections 2024-09-16T01:13:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.205+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2333218760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57e8105ea0 0x7f57e8106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.205+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2333218760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57e8101390 0x7f57e8105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.205+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2333218760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8100e50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.205+0000 7f57edf4b640 1 -- 172.21.15.78:0/2333218760 >> 172.21.15.78:0/2333218760 conn(0x7f57e8077990 msgr2=0x7f57e80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.211+0000 7f57edf4b640 1 -- 172.21.15.78:0/2333218760 shutdown_connections 2024-09-16T01:13:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.214+0000 7f57edf4b640 1 -- 172.21.15.78:0/2333218760 wait complete. 2024-09-16T01:13:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.215+0000 7f57edf4b640 1 Processor -- start 2024-09-16T01:13:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 -- start start 2024-09-16T01:13:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57e8101390 0x7f57e80730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57e8105ea0 0x7f57e810d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f57e81098e0 con 0x7f57e8100a70 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f57e8109760 con 0x7f57e8105ea0 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57edf4b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f57e8109a60 con 0x7f57e8101390 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58500/0 (socket says 172.21.15.78:58500) 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.220+0000 7f57e77fe640 1 -- 172.21.15.78:0/2888886664 learned_addr learned my addr 172.21.15.78:0/2888886664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 -- 172.21.15.78:0/2888886664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57e8101390 msgr2=0x7f57e80730e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57e8101390 0x7f57e80730e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 -- 172.21.15.78:0/2888886664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57e8105ea0 msgr2=0x7f57e810d170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57e8105ea0 0x7f57e810d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 -- 172.21.15.78:0/2888886664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e810d6b0 con 0x7f57e8100a70 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.221+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f57e0000f80 tx=0x7f57e0047380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.222+0000 7f57e4ff9640 1 -- 172.21.15.78:0/2888886664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0044050 con 0x7f57e8100a70 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.222+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e810d8e0 con 0x7f57e8100a70 2024-09-16T01:13:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.222+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e810de30 con 0x7f57e8100a70 2024-09-16T01:13:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.225+0000 7f57e4ff9640 1 -- 172.21.15.78:0/2888886664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e0003e50 con 0x7f57e8100a70 2024-09-16T01:13:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.225+0000 7f57e4ff9640 1 -- 172.21.15.78:0/2888886664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0049690 con 0x7f57e8100a70 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.225+0000 7f57f0dc7740 1 Processor -- start 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 -- start start 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f63e8172f0 con 0x55f63e8b3410 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f63e7bf270 con 0x55f63e8b2af0 2024-09-16T01:13:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.226+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f63e8132b0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.227+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.227+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41024/0 (socket says 172.21.15.78:41024) 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.227+0000 7f57e7fff640 1 -- 172.21.15.78:0/4243582479 learned_addr learned my addr 172.21.15.78:0/4243582479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.229+0000 7f57e77fe640 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 -- 172.21.15.78:0/4243582479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 msgr2=0x55f63e8c00a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8c00a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 -- 172.21.15.78:0/4243582479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 msgr2=0x55f63e8bd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8bd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e4ff9640 1 -- 172.21.15.78:0/2888886664 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f57e0047070 con 0x7f57e8100a70 2024-09-16T01:13:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 -- 172.21.15.78:0/4243582479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f63e7be340 con 0x55f63e8b2af0 2024-09-16T01:13:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57e7fff640 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0096f0 tx=0x7f57dc009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5cf75ac973200a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.230+0000 7f57cdffb640 1 -- 172.21.15.78:0/4243582479 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc015020 con 0x55f63e8b2af0 2024-09-16T01:13:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.232+0000 7f57cdffb640 1 -- 172.21.15.78:0/4243582479 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc00bdb0 con 0x55f63e8b2af0 2024-09-16T01:13:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.233+0000 7f57f0dc7740 1 -- 172.21.15.78:0/4243582479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 msgr2=0x55f63e8b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.233+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0096f0 tx=0x7f57dc009bb0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.233+0000 7f57e4ff9640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 0x7f57d00c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.233+0000 7f57e4ff9640 1 -- 172.21.15.78:0/2888886664 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57e00f3dc0 con 0x7f57e8100a70 2024-09-16T01:13:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.234+0000 7f57e7fff640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 0x7f57d00c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57e7fff640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 0x7f57d00c7b50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0096f0 tx=0x7f57dc0093e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57f0dc7740 1 -- 172.21.15.78:0/4243582479 shutdown_connections 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/4243582479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b2af0 0x55f63e8b2ed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.235+0000 7f57f0dc7740 1 -- 172.21.15.78:0/4243582479 >> 172.21.15.78:0/4243582479 conn(0x55f63e8af060 msgr2=0x55f63e8af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.238+0000 7f57f0dc7740 1 -- 172.21.15.78:0/4243582479 shutdown_connections 2024-09-16T01:13:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.239+0000 7f57f0dc7740 1 -- 172.21.15.78:0/4243582479 wait complete. 2024-09-16T01:13:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.239+0000 7f57f0dc7740 1 -- start start 2024-09-16T01:13:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.239+0000 7f57f0dc7740 1 Processor -- start 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8baa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8baf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f63e8e2930 con 0x55f63e8b3410 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f63e7a0290 con 0x55f63e8bdcb0 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f63e8e2ab0 con 0x55f63e8b2af0 2024-09-16T01:13:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57f0dc7740 10 client.0 _ll_register_callbacks cb 0x55f63e7d1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41034/0 (socket says 172.21.15.78:41034) 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.240+0000 7f57e7fff640 1 -- 172.21.15.78:0/3904193025 learned_addr learned my addr 172.21.15.78:0/3904193025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e7fff640 1 -- 172.21.15.78:0/3904193025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 msgr2=0x55f63e8baa50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8baa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e7fff640 1 -- 172.21.15.78:0/3904193025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 msgr2=0x55f63e8baf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:13:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e6ffd640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8baf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8baf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.241+0000 7f57e7fff640 1 -- 172.21.15.78:0/3904193025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f63e52a210 con 0x55f63e8bdcb0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.242+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0092f0 tx=0x7f57dc009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28bd81c3c02f3a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.242+0000 7f57c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.242+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc015040 con 0x55f63e8bdcb0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.242+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57dc00bdb0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.242+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57dc010cb0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57dc038050 con 0x55f63e8bdcb0 2024-09-16T01:13:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 10 client.25226 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f63e5f6620 con 0x55f63e8bdcb0 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc00bdb0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f57dc01ac30 con 0x55f63e8bdcb0 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 10 client.25226 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 10 client.25226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 20 client.25226 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e8f0120 0x55f63e8f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 4 client.25226 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.243+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f63e824a20 con 0x55f63e8f0120 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.244+0000 7f57e77fe640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e8f0120 0x55f63e8f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.244+0000 7f57e77fe640 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e8f0120 0x55f63e8f25e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd8e4629d92147b server_cookie=e53d284a0f4f40ab in_seq=0 out_seq=0 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.244+0000 7f57c6ffd640 10 client.25226 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3904193025 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f63e824a20 con 0x55f63e8f0120 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57c6ffd640 10 client.25226 handle_command_reply: tid=0 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 client.25226 shutdown 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 msgr2=0x55f63e8bb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0092f0 tx=0x7f57dc009bb0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 shutdown_connections 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e8f0120 0x55f63e8f25e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bb4d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8baf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3904193025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8baa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 >> 172.21.15.78:0/3904193025 conn(0x55f63e8af060 msgr2=0x55f63e8b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57c6ffd640 0 client.25226 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 shutdown_connections 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.245+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3904193025 wait complete. 2024-09-16T01:13:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.246+0000 7f57f0dc7740 20 client.25226 trim_cache size 0 max 16384 2024-09-16T01:13:42.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 Processor -- start 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 -- start start 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 0x55f63e8f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f57d40128c0 con 0x55f63e8b2af0 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f57d4010190 con 0x55f63e8b3410 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f57d400dfe0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e6ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 0x55f63e8f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58522/0 (socket says 172.21.15.78:58522) 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e6ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 0x55f63e8f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41044/0 (socket says 172.21.15.78:41044) 2024-09-16T01:13:42.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 -- 172.21.15.78:0/2695237735 learned_addr learned my addr 172.21.15.78:0/2695237735 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 -- 172.21.15.78:0/2695237735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 msgr2=0x55f63e8b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 -- 172.21.15.78:0/2695237735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 msgr2=0x55f63e8f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 0x55f63e8f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.268+0000 7f57e77fe640 1 -- 172.21.15.78:0/2695237735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f63e8aee90 con 0x55f63e8b2af0 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57e77fe640 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f57e003bfa0 tx=0x7f57e001a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedae96930bc4c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57cd7fa640 1 -- 172.21.15.78:0/2695237735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e005ae60 con 0x55f63e8b2af0 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57cd7fa640 1 -- 172.21.15.78:0/2695237735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e00c03b0 con 0x55f63e8b2af0 2024-09-16T01:13:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57cd7fa640 1 -- 172.21.15.78:0/2695237735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0047ab0 con 0x55f63e8b2af0 2024-09-16T01:13:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57f0dc7740 1 -- 172.21.15.78:0/2695237735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 msgr2=0x55f63e8f7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f57e003bfa0 tx=0x7f57e001a5d0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57f0dc7740 1 -- 172.21.15.78:0/2695237735 shutdown_connections 2024-09-16T01:13:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.269+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8bdcb0 0x55f63e8b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8b3410 0x55f63e8f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/2695237735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b2af0 0x55f63e8f7ef0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 -- 172.21.15.78:0/2695237735 >> 172.21.15.78:0/2695237735 conn(0x55f63e8af060 msgr2=0x55f63e8b9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 -- 172.21.15.78:0/2695237735 shutdown_connections 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 -- 172.21.15.78:0/2695237735 wait complete. 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 -- start start 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 Processor -- start 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8be190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8be6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.270+0000 7f57f0dc7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57f0dc7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f57d400e650 con 0x55f63e8b3410 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8be190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8be190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58698/0 (socket says 172.21.15.78:58698) 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57e77fe640 1 -- 172.21.15.78:0/3464437988 learned_addr learned my addr 172.21.15.78:0/3464437988 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f57d400fac0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f57d400ed20 con 0x55f63e8b2af0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57f0dc7740 10 client.0 _ll_register_callbacks cb 0x7f57dc009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.271+0000 7f57e6ffd640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8be6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 -- 172.21.15.78:0/3464437988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 msgr2=0x55f63e8be190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8be190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 -- 172.21.15.78:0/3464437988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 msgr2=0x55f63e8be6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8be6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f63e8e70e0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57e7fff640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bec30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00b4b0 tx=0x7f57dc010640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615596b2eede1d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:13:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc027bf0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f57dc027d90 con 0x55f63e8bdcb0 2024-09-16T01:13:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.272+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e00c03b0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57dc049050 con 0x55f63e8bdcb0 2024-09-16T01:13:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 10 client.25232 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f63e8e83d0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e00c03b0 con 0x55f63e8bdcb0 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f57dc044030 con 0x55f63e8bdcb0 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 10 client.25232 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 10 client.25232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 20 client.25232 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e95aec0 0x55f63e95d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.273+0000 7f57f0dc7740 4 client.25232 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.274+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f63e81ed70 con 0x55f63e95aec0 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.274+0000 7f57e77fe640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e95aec0 0x55f63e95d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.274+0000 7f57c6ffd640 10 client.25232 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.274+0000 7f57e77fe640 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e95aec0 0x55f63e95d2b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5d0086e627677a9 server_cookie=f5f7e69774abfcbb in_seq=0 out_seq=0 2024-09-16T01:13:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57c6ffd640 1 -- 172.21.15.78:0/3464437988 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f63e81ed70 con 0x55f63e95aec0 2024-09-16T01:13:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57c6ffd640 10 client.25232 handle_command_reply: tid=0 2024-09-16T01:13:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 client.25232 shutdown 2024-09-16T01:13:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 msgr2=0x55f63e8bec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bec30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00b4b0 tx=0x7f57dc010640 comp rx=0 tx=0).stop 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 shutdown_connections 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f63e95aec0 0x55f63e95d2b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f63e8bdcb0 0x55f63e8bec30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f63e8b3410 0x55f63e8be6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57c6ffd640 0 client.25232 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 --2- 172.21.15.78:0/3464437988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f63e8b2af0 0x55f63e8be190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.281+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 >> 172.21.15.78:0/3464437988 conn(0x55f63e8af060 msgr2=0x55f63e8b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.282+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 shutdown_connections 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.282+0000 7f57f0dc7740 1 -- 172.21.15.78:0/3464437988 wait complete. 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.282+0000 7f57f0dc7740 20 client.25232 trim_cache size 0 max 16384 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:13:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:13:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 msgr2=0x7f57d00c7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 0x7f57d00c7b50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0096f0 tx=0x7f57dc0093e0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 msgr2=0x7f57e8072ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f57e0000f80 tx=0x7f57e0047380 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 shutdown_connections 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57d00c5690 0x7f57d00c7b50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57e8105ea0 0x7f57e810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57e8101390 0x7f57e80730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 --2- 172.21.15.78:0/2888886664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57e8100a70 0x7f57e8072ba0 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f57e0000f80 tx=0x7f57e0047380 comp rx=0 tx=0).stop 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 >> 172.21.15.78:0/2888886664 conn(0x7f57e8077990 msgr2=0x7f57e8103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 shutdown_connections 2024-09-16T01:13:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:42.283+0000 7f57edf4b640 1 -- 172.21.15.78:0/2888886664 wait complete. 2024-09-16T01:13:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:42 smithi078 ceph-mon[30937]: pgmap v90: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-09-16T01:13:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:42 smithi193 ceph-mon[35701]: pgmap v90: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-09-16T01:13:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:42 smithi148 ceph-mon[35030]: pgmap v90: 129 pgs: 129 active+clean; 1.5 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-09-16T01:13:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:44 smithi078 ceph-mon[30937]: pgmap v91: 129 pgs: 129 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-09-16T01:13:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:44 smithi193 ceph-mon[35701]: pgmap v91: 129 pgs: 129 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-09-16T01:13:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:44 smithi148 ceph-mon[35030]: pgmap v91: 129 pgs: 129 active+clean; 1.5 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-09-16T01:13:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 150.0 s, 2569.8 tps, lat 42.144 ms stddev 240.091 2024-09-16T01:13:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:46 smithi078 ceph-mon[30937]: pgmap v92: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:13:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:46 smithi193 ceph-mon[35701]: pgmap v92: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:13:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:46 smithi148 ceph-mon[35030]: pgmap v92: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:13:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:48 smithi078 ceph-mon[30937]: pgmap v93: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-09-16T01:13:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:48 smithi193 ceph-mon[35701]: pgmap v93: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-09-16T01:13:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:48 smithi148 ceph-mon[35030]: pgmap v93: 129 pgs: 129 active+clean; 1.5 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-09-16T01:13:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 155.0 s, 2326.4 tps, lat 42.694 ms stddev 48.325 2024-09-16T01:13:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:50 smithi078 ceph-mon[30937]: pgmap v94: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.6 MiB/s wr, 695 op/s 2024-09-16T01:13:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:50 smithi193 ceph-mon[35701]: pgmap v94: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.6 MiB/s wr, 695 op/s 2024-09-16T01:13:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:50 smithi148 ceph-mon[35030]: pgmap v94: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.6 MiB/s wr, 695 op/s 2024-09-16T01:13:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:52 smithi078 ceph-mon[30937]: pgmap v95: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.4 MiB/s wr, 437 op/s 2024-09-16T01:13:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:52 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:13:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:52 smithi193 ceph-mon[35701]: pgmap v95: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.4 MiB/s wr, 437 op/s 2024-09-16T01:13:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:52 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:13:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:52 smithi148 ceph-mon[35030]: pgmap v95: 129 pgs: 129 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.4 MiB/s wr, 437 op/s 2024-09-16T01:13:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:52 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:13:54.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:13:54.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:13:54.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:53 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:13:54.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:13:54.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:13:54.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:53 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:13:54.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:13:54.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:13:54.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:53 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:13:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:54 smithi078 ceph-mon[30937]: pgmap v96: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.0 MiB/s wr, 605 op/s 2024-09-16T01:13:55.286 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:54 smithi148 ceph-mon[35030]: pgmap v96: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.0 MiB/s wr, 605 op/s 2024-09-16T01:13:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:54 smithi193 ceph-mon[35701]: pgmap v96: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.0 MiB/s wr, 605 op/s 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 -- 172.21.15.78:0/2614691162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f010c0d0 msgr2=0x7f88f010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 --2- 172.21.15.78:0/2614691162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f010c0d0 0x7f88f010e570 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009ca0 tx=0x7f88e802ecd0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 -- 172.21.15.78:0/2614691162 shutdown_connections 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 --2- 172.21.15.78:0/2614691162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f010c0d0 0x7f88f010e570 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 --2- 172.21.15.78:0/2614691162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f01035c0 0x7f88f010baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 --2- 172.21.15.78:0/2614691162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f0102ca0 0x7f88f0103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 -- 172.21.15.78:0/2614691162 >> 172.21.15.78:0/2614691162 conn(0x7f88f00fc6b0 msgr2=0x7f88f00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 -- 172.21.15.78:0/2614691162 shutdown_connections 2024-09-16T01:13:55.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.420+0000 7f88f708d640 1 -- 172.21.15.78:0/2614691162 wait complete. 2024-09-16T01:13:55.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 Processor -- start 2024-09-16T01:13:55.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 -- start start 2024-09-16T01:13:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 0x7f88f01059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 0x7f88f0109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88f006c390 con 0x7f88f0102ca0 2024-09-16T01:13:55.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88f006c210 con 0x7f88f010c0d0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f708d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88f006c510 con 0x7f88f01035c0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56800/0 (socket says 172.21.15.78:56800) 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f4e29640 1 -- 172.21.15.78:0/2304778350 learned_addr learned my addr 172.21.15.78:0/2304778350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88effff640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 0x7f88f01059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.421+0000 7f88f562a640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 0x7f88f0109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 -- 172.21.15.78:0/2304778350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 msgr2=0x7f88f0109dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 0x7f88f0109dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 -- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 msgr2=0x7f88f0105490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 -- 172.21.15.78:0/2304778350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f010a4f0 con 0x7f88f01035c0 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88f4e29640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88f562a640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 0x7f88f0109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:55.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88effff640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 0x7f88f01059d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00e9c0 tx=0x7f88dc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88edffb640 1 -- 172.21.15.78:0/2304778350 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00ccc0 con 0x7f88f01035c0 2024-09-16T01:13:55.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88edffb640 1 -- 172.21.15.78:0/2304778350 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc0035f0 con 0x7f88f01035c0 2024-09-16T01:13:55.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f010a7e0 con 0x7f88f01035c0 2024-09-16T01:13:55.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.422+0000 7f88edffb640 1 -- 172.21.15.78:0/2304778350 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00ce60 con 0x7f88f01035c0 2024-09-16T01:13:55.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.423+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01b3b10 con 0x7f88f01035c0 2024-09-16T01:13:55.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.426+0000 7f88f9e2c740 1 Processor -- start 2024-09-16T01:13:55.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 -- start start 2024-09-16T01:13:55.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a22af0 0x55ff31a22ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a2dcb0 0x55ff31a300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ff319872f0 con 0x55ff31a2dcb0 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ff3192f270 con 0x55ff31a23410 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff319832b0 con 0x55ff31a22af0 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:42010/0 (socket says 172.21.15.78:42010) 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 -- 172.21.15.78:0/1047378544 learned_addr learned my addr 172.21.15.78:0/1047378544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f562a640 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a22af0 0x55ff31a22ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88edffb640 1 -- 172.21.15.78:0/2304778350 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f88dc0037a0 con 0x7f88f01035c0 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 -- 172.21.15.78:0/1047378544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a22af0 msgr2=0x55ff31a22ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a22af0 0x55ff31a22ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 -- 172.21.15.78:0/1047378544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a2dcb0 msgr2=0x55ff31a300a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:13:55.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a2dcb0 0x55ff31a300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.427+0000 7f88f4e29640 1 -- 172.21.15.78:0/1047378544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff3192e340 con 0x55ff31a23410 2024-09-16T01:13:55.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88f4e29640 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88e4011630 tx=0x7f88e4011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ecba8cd5e40b79d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88daffd640 1 -- 172.21.15.78:0/1047378544 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e401e070 con 0x55ff31a23410 2024-09-16T01:13:55.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88daffd640 1 -- 172.21.15.78:0/1047378544 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e400fbd0 con 0x55ff31a23410 2024-09-16T01:13:55.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88edffb640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 0x7f88e00a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88edffb640 1 -- 172.21.15.78:0/2304778350 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f88dc0d03d0 con 0x7f88f01035c0 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.428+0000 7f88f562a640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 0x7f88e00a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88daffd640 1 -- 172.21.15.78:0/1047378544 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4019d80 con 0x55ff31a23410 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 -- 172.21.15.78:0/1047378544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 msgr2=0x55ff31a2d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88e4011630 tx=0x7f88e4011af0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f562a640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 0x7f88e00a59f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005e30 tx=0x7f88e8010040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 -- 172.21.15.78:0/1047378544 shutdown_connections 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a2dcb0 0x55ff31a300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a23410 0x55ff31a2d770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/1047378544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a22af0 0x55ff31a22ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.429+0000 7f88f9e2c740 1 -- 172.21.15.78:0/1047378544 >> 172.21.15.78:0/1047378544 conn(0x55ff31a1f060 msgr2=0x55ff31a1f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.430+0000 7f88f9e2c740 1 -- 172.21.15.78:0/1047378544 shutdown_connections 2024-09-16T01:13:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.430+0000 7f88f9e2c740 1 -- 172.21.15.78:0/1047378544 wait complete. 2024-09-16T01:13:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.430+0000 7f88f9e2c740 1 -- start start 2024-09-16T01:13:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 Processor -- start 2024-09-16T01:13:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a2b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a2b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ff31a30920 con 0x55ff31a22af0 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ff31910290 con 0x55ff31a2dcb0 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ff31a30aa0 con 0x55ff31a23410 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f9e2c740 10 client.0 _ll_register_callbacks cb 0x55ff31941140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56828/0 (socket says 172.21.15.78:56828) 2024-09-16T01:13:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f4e29640 1 -- 172.21.15.78:0/2498213059 learned_addr learned my addr 172.21.15.78:0/2498213059 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88f562a640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a2b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.431+0000 7f88effff640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a2b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 -- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 msgr2=0x55ff31a2b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a2b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 -- 172.21.15.78:0/2498213059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 msgr2=0x55ff31a2ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 -- 172.21.15.78:0/2498213059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff3169a210 con 0x55ff31a23410 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88f562a640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a2b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88effff640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a2b080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0cd790 tx=0x7f88dc09ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afe3258393dde41e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88cffff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0a4f90 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e400fbd0 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f88dc098990 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f88dc09b770 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88f9e2c740 10 client.25306 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ff31766620 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.432+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e400fbd0 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f88dc004000 con 0x55ff31a23410 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 10 client.25306 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 10 client.25306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 20 client.25306 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31a3e130 0x55ff31a40550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 4 client.25306 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:13:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff31994a20 con 0x55ff31a3e130 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f4e29640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31a3e130 0x55ff31a40550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88cffff640 10 client.25306 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.433+0000 7f88f4e29640 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31a3e130 0x55ff31a40550 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1572a4215d20edd6 server_cookie=c229d89789f179f2 in_seq=0 out_seq=0 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.434+0000 7f88cffff640 1 -- 172.21.15.78:0/2498213059 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ff31994a20 con 0x55ff31a3e130 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.434+0000 7f88cffff640 10 client.25306 handle_command_reply: tid=0 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.434+0000 7f88f9e2c740 1 client.25306 shutdown 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.434+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 msgr2=0x55ff31a2b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.434+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a2b080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0cd790 tx=0x7f88dc09ca20 comp rx=0 tx=0).stop 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 shutdown_connections 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31a3e130 0x55ff31a40550 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a2b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a2b080 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2498213059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 >> 172.21.15.78:0/2498213059 conn(0x55ff31a1f060 msgr2=0x55ff31a24d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.435+0000 7f88cffff640 0 client.25306 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.436+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 shutdown_connections 2024-09-16T01:13:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.436+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2498213059 wait complete. 2024-09-16T01:13:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.436+0000 7f88f9e2c740 20 client.25306 trim_cache size 0 max 16384 2024-09-16T01:13:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.454+0000 7f88f9e2c740 1 Processor -- start 2024-09-16T01:13:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.454+0000 7f88f9e2c740 1 -- start start 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a46360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01c0640 con 0x55ff31a22af0 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01bdf10 con 0x55ff31a2dcb0 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01bbd60 con 0x55ff31a23410 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54732/0 (socket says 172.21.15.78:54732) 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 -- 172.21.15.78:0/2342300334 learned_addr learned my addr 172.21.15.78:0/2342300334 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88f562a640 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a46360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 -- 172.21.15.78:0/2342300334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 msgr2=0x55ff31a46360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a46360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 -- 172.21.15.78:0/2342300334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 msgr2=0x55ff31a2b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.455+0000 7f88effff640 1 -- 172.21.15.78:0/2342300334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff31a25940 con 0x55ff31a23410 2024-09-16T01:13:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.456+0000 7f88effff640 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0a4a60 tx=0x7f88dc0044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4dd74277a897a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.456+0000 7f88da7fc640 1 -- 172.21.15.78:0/2342300334 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc09be60 con 0x55ff31a23410 2024-09-16T01:13:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.456+0000 7f88da7fc640 1 -- 172.21.15.78:0/2342300334 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc0a4f90 con 0x55ff31a23410 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.456+0000 7f88da7fc640 1 -- 172.21.15.78:0/2342300334 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc0b50f0 con 0x55ff31a23410 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2342300334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 msgr2=0x55ff31a45e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0a4a60 tx=0x7f88dc0044b0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2342300334 shutdown_connections 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a46360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a45e20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/2342300334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2342300334 >> 172.21.15.78:0/2342300334 conn(0x55ff31a1f060 msgr2=0x55ff31a2cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2342300334 shutdown_connections 2024-09-16T01:13:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- 172.21.15.78:0/2342300334 wait complete. 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 -- start start 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.457+0000 7f88f9e2c740 1 Processor -- start 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a36560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01bc3d0 con 0x55ff31a22af0 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01bd840 con 0x55ff31a2dcb0 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01bcaa0 con 0x55ff31a23410 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f9e2c740 10 client.0 _ll_register_callbacks cb 0x7f88dc004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:13:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f4e29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56846/0 (socket says 172.21.15.78:56846) 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f4e29640 1 -- 172.21.15.78:0/34299635 learned_addr learned my addr 172.21.15.78:0/34299635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88effff640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a36560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.458+0000 7f88f562a640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 -- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 msgr2=0x55ff31a274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 -- 172.21.15.78:0/34299635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 msgr2=0x55ff31a36020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 -- 172.21.15.78:0/34299635 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff31a27ca0 con 0x55ff31a23410 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88f4e29640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88effff640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a36560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0cdd90 tx=0x7f88dc098640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=763be875d54755a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88cffff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:13:55.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc09be60 con 0x55ff31a23410 2024-09-16T01:13:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f88dc0bc0d0 con 0x55ff31a23410 2024-09-16T01:13:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.459+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88dc0a4f90 con 0x55ff31a23410 2024-09-16T01:13:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.460+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f88dc0b52f0 con 0x55ff31a23410 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.460+0000 7f88f9e2c740 10 client.25312 fetch_fsmap learned FSMap version 30 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.460+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ff31a27ca0 con 0x55ff31a23410 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88dc0bc2d0 con 0x55ff31a23410 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f88dc0a02a0 con 0x55ff31a23410 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 10 client.25312 fetch_fsmap finished waiting for FSMap version 30 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 10 client.25312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 20 client.25312 populate_metadata read hostname 'smithi078' 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31aa8c10 0x55ff31aab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 4 client.25312 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.461+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f88dc0a4f90 con 0x55ff31aa8c10 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.462+0000 7f88f4e29640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31aa8c10 0x55ff31aab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:13:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.462+0000 7f88cffff640 10 client.25312 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.462+0000 7f88f4e29640 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31aa8c10 0x55ff31aab030 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=affaf8b64bd409f5 server_cookie=327e046d94b2073e in_seq=0 out_seq=0 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88cffff640 1 -- 172.21.15.78:0/34299635 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f88dc0a4f90 con 0x55ff31aa8c10 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88cffff640 10 client.25312 handle_command_reply: tid=0 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 client.25312 shutdown 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 msgr2=0x55ff31a36560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a36560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0cdd90 tx=0x7f88dc098640 comp rx=0 tx=0).stop 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 shutdown_connections 2024-09-16T01:13:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ff31aa8c10 0x55ff31aab030 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff31a2dcb0 0x55ff31a274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ff31a23410 0x55ff31a36560 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88cffff640 0 client.25312 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 --2- 172.21.15.78:0/34299635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff31a22af0 0x55ff31a36020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.468+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 >> 172.21.15.78:0/34299635 conn(0x55ff31a1f060 msgr2=0x55ff31a1d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 shutdown_connections 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f9e2c740 1 -- 172.21.15.78:0/34299635 wait complete. 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f9e2c740 20 client.25312 trim_cache size 0 max 16384 2024-09-16T01:13:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:13:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:13:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:13:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:13:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 msgr2=0x7f88e00a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 0x7f88e00a59f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88e8005e30 tx=0x7f88e8010040 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 msgr2=0x7f88f01059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.469+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 0x7f88f01059d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00e9c0 tx=0x7f88dc00ee80 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 shutdown_connections 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f88e00a3530 0x7f88e00a59f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88f010c0d0 0x7f88f0109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88f01035c0 0x7f88f01059d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 --2- 172.21.15.78:0/2304778350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88f0102ca0 0x7f88f0105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 >> 172.21.15.78:0/2304778350 conn(0x7f88f00fc6b0 msgr2=0x7f88f010da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 shutdown_connections 2024-09-16T01:13:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:13:55.470+0000 7f88f708d640 1 -- 172.21.15.78:0/2304778350 wait complete. 2024-09-16T01:13:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 160.0 s, 2690.8 tps, lat 37.240 ms stddev 42.024 2024-09-16T01:13:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:56 smithi078 ceph-mon[30937]: pgmap v97: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.2 MiB/s wr, 748 op/s 2024-09-16T01:13:57.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:56 smithi193 ceph-mon[35701]: pgmap v97: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.2 MiB/s wr, 748 op/s 2024-09-16T01:13:57.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:56 smithi148 ceph-mon[35030]: pgmap v97: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.2 MiB/s wr, 748 op/s 2024-09-16T01:13:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:13:58 smithi078 ceph-mon[30937]: pgmap v98: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 643 op/s 2024-09-16T01:13:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:13:58 smithi193 ceph-mon[35701]: pgmap v98: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 643 op/s 2024-09-16T01:13:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:13:58 smithi148 ceph-mon[35030]: pgmap v98: 129 pgs: 129 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 MiB/s wr, 643 op/s 2024-09-16T01:14:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 165.0 s, 1795.6 tps, lat 54.722 ms stddev 66.062 2024-09-16T01:14:01.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:01 smithi078 ceph-mon[30937]: pgmap v99: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 958 op/s 2024-09-16T01:14:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:01 smithi193 ceph-mon[35701]: pgmap v99: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 958 op/s 2024-09-16T01:14:01.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:01 smithi148 ceph-mon[35030]: pgmap v99: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 958 op/s 2024-09-16T01:14:03.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:03 smithi078 ceph-mon[30937]: pgmap v100: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 MiB/s wr, 667 op/s 2024-09-16T01:14:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:03 smithi193 ceph-mon[35701]: pgmap v100: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 MiB/s wr, 667 op/s 2024-09-16T01:14:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:03 smithi148 ceph-mon[35030]: pgmap v100: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.1 MiB/s wr, 667 op/s 2024-09-16T01:14:04.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:04 smithi078 ceph-mon[30937]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:14:04.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:04 smithi193 ceph-mon[35701]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:14:04.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:04 smithi148 ceph-mon[35030]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-09-16T01:14:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:05 smithi193 ceph-mon[35701]: pgmap v101: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-09-16T01:14:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:05 smithi148 ceph-mon[35030]: pgmap v101: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-09-16T01:14:05.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:05 smithi078 ceph-mon[30937]: pgmap v101: 129 pgs: 129 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.9 MiB/s wr, 733 op/s 2024-09-16T01:14:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 170.0 s, 488.6 tps, lat 207.398 ms stddev 461.311 2024-09-16T01:14:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:07 smithi193 ceph-mon[35701]: pgmap v102: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-09-16T01:14:07.330 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:07 smithi078 ceph-mon[30937]: pgmap v102: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-09-16T01:14:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:07 smithi148 ceph-mon[35030]: pgmap v102: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-09-16T01:14:08.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 -- 172.21.15.78:0/2688934489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc41013b0 msgr2=0x7fdbc4105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 --2- 172.21.15.78:0/2688934489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc41013b0 0x7fdbc4105870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009860 tx=0x7fdbb802a340 comp rx=0 tx=0).stop 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 -- 172.21.15.78:0/2688934489 shutdown_connections 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 --2- 172.21.15.78:0/2688934489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc4105f10 0x7fdbc41063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 --2- 172.21.15.78:0/2688934489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc41013b0 0x7fdbc4105870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 --2- 172.21.15.78:0/2688934489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.598+0000 7fdbca959640 1 -- 172.21.15.78:0/2688934489 >> 172.21.15.78:0/2688934489 conn(0x7fdbc40fc6b0 msgr2=0x7fdbc40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.599+0000 7fdbca959640 1 -- 172.21.15.78:0/2688934489 shutdown_connections 2024-09-16T01:14:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.599+0000 7fdbca959640 1 -- 172.21.15.78:0/2688934489 wait complete. 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 Processor -- start 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 -- start start 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 0x7fdbc4073e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc4105f10 0x7fdbc406cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.600+0000 7fdbca959640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc4118500 con 0x7fdbc41013b0 2024-09-16T01:14:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbca959640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc4118380 con 0x7fdbc4105f10 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbca959640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc4118680 con 0x7fdbc4100a90 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 0x7fdbc4073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36934/0 (socket says 172.21.15.78:36934) 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 -- 172.21.15.78:0/834404987 learned_addr learned my addr 172.21.15.78:0/834404987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc4105f10 0x7fdbc406cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 -- 172.21.15.78:0/834404987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc4105f10 msgr2=0x7fdbc406cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc4105f10 0x7fdbc406cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 -- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 msgr2=0x7fdbc4073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 0x7fdbc4073e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc3fff640 1 -- 172.21.15.78:0/834404987 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc406d420 con 0x7fdbc4100a90 2024-09-16T01:14:08.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.601+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 0x7fdbc4073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00efb0 tx=0x7fdbbc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbc17fa640 1 -- 172.21.15.78:0/834404987 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc01b070 con 0x7fdbc4100a90 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbc17fa640 1 -- 172.21.15.78:0/834404987 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbbc00ba50 con 0x7fdbc4100a90 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbc17fa640 1 -- 172.21.15.78:0/834404987 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc016c00 con 0x7fdbc4100a90 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc406d710 con 0x7fdbc4100a90 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.602+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc4071710 con 0x7fdbc4100a90 2024-09-16T01:14:08.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 Processor -- start 2024-09-16T01:14:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 -- start start 2024-09-16T01:14:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e8f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e903770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55714e85d2f0 con 0x55714e903cb0 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55714e805270 con 0x55714e8f8af0 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.606+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55714e8592b0 con 0x55714e8f9410 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e8f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49450/0 (socket says 172.21.15.78:49450) 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 -- 172.21.15.78:0/2144526426 learned_addr learned my addr 172.21.15.78:0/2144526426 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e8f8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:54346/0 (socket says 172.21.15.78:54346) 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 -- 172.21.15.78:0/2144526426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 msgr2=0x55714e903770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e903770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 -- 172.21.15.78:0/2144526426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 msgr2=0x55714e8f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e8f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc37fe640 1 -- 172.21.15.78:0/2144526426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714e804340 con 0x55714e903cb0 2024-09-16T01:14:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.607+0000 7fdbc17fa640 1 -- 172.21.15.78:0/834404987 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fdbbc01e070 con 0x7fdbc4100a90 2024-09-16T01:14:08.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.608+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009780 tx=0x7fdbb80095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec41ee6e084ead3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.608+0000 7fdbc17fa640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 0x7fdbac0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.609+0000 7fdbc17fa640 1 -- 172.21.15.78:0/834404987 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdbbc0cffe0 con 0x7fdbc4100a90 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.609+0000 7fdbaa7fc640 1 -- 172.21.15.78:0/2144526426 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8041020 con 0x55714e903cb0 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.609+0000 7fdbaa7fc640 1 -- 172.21.15.78:0/2144526426 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb8030840 con 0x55714e903cb0 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.609+0000 7fdbaa7fc640 1 -- 172.21.15.78:0/2144526426 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8037040 con 0x55714e903cb0 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 0x7fdbac0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/2144526426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 msgr2=0x55714e9060a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009780 tx=0x7fdbb80095a0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/2144526426 shutdown_connections 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e9060a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e903770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/2144526426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e8f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/2144526426 >> 172.21.15.78:0/2144526426 conn(0x55714e8f5060 msgr2=0x55714e8f5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/2144526426 shutdown_connections 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.610+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/2144526426 wait complete. 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.611+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 0x7fdbac0a59a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009a70 tx=0x7fdbb40090e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.611+0000 7fdbcd7d0740 1 -- start start 2024-09-16T01:14:08.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.611+0000 7fdbcd7d0740 1 Processor -- start 2024-09-16T01:14:08.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f8af0 0x55714e900c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 0x55714e901180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55714e906880 con 0x55714e8f9410 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55714e7e6290 con 0x55714e903cb0 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55714e906a00 con 0x55714e8f8af0 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbcd7d0740 10 client.0 _ll_register_callbacks cb 0x55714e817140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:54360/0 (socket says 172.21.15.78:54360) 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/3879809563 learned_addr learned my addr 172.21.15.78:0/3879809563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/3879809563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f8af0 msgr2=0x55714e900c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 0x55714e901180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f8af0 0x55714e900c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/3879809563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 msgr2=0x55714e901180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.612+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 0x55714e901180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.613+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/3879809563 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714e570210 con 0x55714e903cb0 2024-09-16T01:14:08.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.613+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 0x55714e901180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:14:08.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.613+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4011e70 tx=0x7fdbb40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa3493f4baecf69b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.613+0000 7fdba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:14:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.614+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4047070 con 0x55714e903cb0 2024-09-16T01:14:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.614+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdbb4043610 con 0x55714e903cb0 2024-09-16T01:14:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.614+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb8030840 con 0x55714e903cb0 2024-09-16T01:14:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbb4051050 con 0x55714e903cb0 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdbcd7d0740 10 client.25262 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55714e63c620 con 0x55714e903cb0 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb8030840 con 0x55714e903cb0 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdbb404c9d0 con 0x55714e903cb0 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdbcd7d0740 10 client.25262 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdbcd7d0740 10 client.25262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.615+0000 7fdbcd7d0740 20 client.25262 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.616+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e914180 0x55714e916640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.616+0000 7fdbcd7d0740 4 client.25262 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:14:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.616+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55714e86aa20 con 0x55714e914180 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.616+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e914180 0x55714e916640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.617+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e914180 0x55714e916640 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e099f9abb74e4ae server_cookie=c7364c92dee476ad in_seq=0 out_seq=0 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.617+0000 7fdba37fe640 10 client.25262 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.617+0000 7fdba37fe640 1 -- 172.21.15.78:0/3879809563 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55714e86aa20 con 0x55714e914180 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.617+0000 7fdba37fe640 10 client.25262 handle_command_reply: tid=0 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.617+0000 7fdbcd7d0740 1 client.25262 shutdown 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 msgr2=0x55714e9016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4011e70 tx=0x7fdbb40046c0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 shutdown_connections 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e914180 0x55714e916640 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e903cb0 0x55714e9016c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f9410 0x55714e901180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/3879809563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f8af0 0x55714e900c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 >> 172.21.15.78:0/3879809563 conn(0x55714e8f5060 msgr2=0x55714e8fad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdba37fe640 0 client.25262 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 shutdown_connections 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/3879809563 wait complete. 2024-09-16T01:14:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.618+0000 7fdbcd7d0740 20 client.25262 trim_cache size 0 max 16384 2024-09-16T01:14:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.639+0000 7fdbcd7d0740 1 Processor -- start 2024-09-16T01:14:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 -- start start 2024-09-16T01:14:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 0x55714e902bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f9410 0x55714e9014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc41a0e70 con 0x55714e8f8af0 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc419e740 con 0x55714e8f9410 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc4105870 con 0x55714e903cb0 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36954/0 (socket says 172.21.15.78:36954) 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/1372607339 learned_addr learned my addr 172.21.15.78:0/1372607339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 0x55714e902bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.640+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f9410 0x55714e9014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/1372607339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f9410 msgr2=0x55714e9014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f9410 0x55714e9014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/1372607339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 msgr2=0x55714e902bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 0x55714e902bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/1372607339 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714e8f61d0 con 0x55714e903cb0 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 0x55714e902bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdbc8ef6640 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4011730 tx=0x7fdbb4011860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13bed331db3bf85c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdba9ffb640 1 -- 172.21.15.78:0/1372607339 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4047040 con 0x55714e903cb0 2024-09-16T01:14:08.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdba9ffb640 1 -- 172.21.15.78:0/1372607339 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb401b070 con 0x55714e903cb0 2024-09-16T01:14:08.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.641+0000 7fdba9ffb640 1 -- 172.21.15.78:0/1372607339 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4043900 con 0x55714e903cb0 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.642+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/1372607339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 msgr2=0x55714e91bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.642+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4011730 tx=0x7fdbb4011860 comp rx=0 tx=0).stop 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/1372607339 shutdown_connections 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e903cb0 0x55714e91bf50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f9410 0x55714e9014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/1372607339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e8f8af0 0x55714e902bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/1372607339 >> 172.21.15.78:0/1372607339 conn(0x55714e8f5060 msgr2=0x55714e9024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/1372607339 shutdown_connections 2024-09-16T01:14:08.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.643+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/1372607339 wait complete. 2024-09-16T01:14:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.644+0000 7fdbcd7d0740 1 -- start start 2024-09-16T01:14:08.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.644+0000 7fdbcd7d0740 1 Processor -- start 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.644+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e90bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.644+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e90c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.644+0000 7fdbcd7d0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e8fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc410fa20 con 0x55714e903cb0 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc419e070 con 0x55714e8f8af0 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbcd7d0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc41100d0 con 0x55714e8f9410 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbcd7d0740 10 client.0 _ll_register_callbacks cb 0x7fdbb4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e8fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc8ef6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e8fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49474/0 (socket says 172.21.15.78:49474) 2024-09-16T01:14:08.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc8ef6640 1 -- 172.21.15.78:0/971515444 learned_addr learned my addr 172.21.15.78:0/971515444 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e90bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 -- 172.21.15.78:0/971515444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 msgr2=0x55714e90c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e90c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 -- 172.21.15.78:0/971515444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 msgr2=0x55714e8fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e8fd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.645+0000 7fdbc3fff640 1 -- 172.21.15.78:0/971515444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714e8fdab0 con 0x55714e8f8af0 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.646+0000 7fdba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.646+0000 7fdbc3fff640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e90bc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0cd3a0 tx=0x7fdbbc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3b29dfcd881c70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.646+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc09a2e0 con 0x55714e8f8af0 2024-09-16T01:14:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.647+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb401b070 con 0x55714e8f8af0 2024-09-16T01:14:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.647+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdbbc0b7070 con 0x55714e8f8af0 2024-09-16T01:14:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.647+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbbc09a2e0 con 0x55714e8f8af0 2024-09-16T01:14:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.648+0000 7fdbcd7d0740 10 client.25268 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.648+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55714e97e920 con 0x55714e8f8af0 2024-09-16T01:14:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.650+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb401b070 con 0x55714e8f8af0 2024-09-16T01:14:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.650+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdbbc00b040 con 0x55714e8f8af0 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 10 client.25268 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 10 client.25268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 20 client.25268 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e97efa0 0x55714e9813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 4 client.25268 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:14:08.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.651+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55714e864d70 con 0x55714e97efa0 2024-09-16T01:14:08.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.655+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e97efa0 0x55714e9813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.658+0000 7fdbc37fe640 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e97efa0 0x55714e9813c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa74729ffc8e883 server_cookie=ff4fcc7aa2b7e494 in_seq=0 out_seq=0 2024-09-16T01:14:08.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.658+0000 7fdba37fe640 10 client.25268 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.706+0000 7fdba37fe640 1 -- 172.21.15.78:0/971515444 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55714e864d70 con 0x55714e97efa0 2024-09-16T01:14:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.706+0000 7fdba37fe640 10 client.25268 handle_command_reply: tid=0 2024-09-16T01:14:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.706+0000 7fdbcd7d0740 1 client.25268 shutdown 2024-09-16T01:14:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 msgr2=0x55714e90bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e90bc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0cd3a0 tx=0x7fdbbc009b50 comp rx=0 tx=0).stop 2024-09-16T01:14:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 shutdown_connections 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55714e97efa0 0x55714e9813c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55714e903cb0 0x55714e8fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55714e8f9410 0x55714e90c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 --2- 172.21.15.78:0/971515444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55714e8f8af0 0x55714e90bc90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 >> 172.21.15.78:0/971515444 conn(0x55714e8f5060 msgr2=0x55714e8fad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.707+0000 7fdba37fe640 0 client.25268 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.708+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 shutdown_connections 2024-09-16T01:14:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.708+0000 7fdbcd7d0740 1 -- 172.21.15.78:0/971515444 wait complete. 2024-09-16T01:14:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.708+0000 7fdbcd7d0740 20 client.25268 trim_cache size 0 max 16384 2024-09-16T01:14:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:14:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:14:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:14:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.713+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 msgr2=0x7fdbac0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.713+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 0x7fdbac0a59a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009a70 tx=0x7fdbb40090e0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 msgr2=0x7fdbc4073950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00efb0 tx=0x7fdbbc00c800 comp rx=0 tx=0).stop 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 shutdown_connections 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdbac0a34e0 0x7fdbac0a59a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc4105f10 0x7fdbc406cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbc41013b0 0x7fdbc4073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 --2- 172.21.15.78:0/834404987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100a90 0x7fdbc4073950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.714+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 >> 172.21.15.78:0/834404987 conn(0x7fdbc40fc6b0 msgr2=0x7fdbc41037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.715+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 shutdown_connections 2024-09-16T01:14:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:08.715+0000 7fdbca959640 1 -- 172.21.15.78:0/834404987 wait complete. 2024-09-16T01:14:09.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:09 smithi193 ceph-mon[35701]: pgmap v103: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:14:09.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:09 smithi148 ceph-mon[35030]: pgmap v103: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:14:09.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:09 smithi078 ceph-mon[30937]: pgmap v103: 129 pgs: 129 active+clean; 1.7 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:14:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 175.0 s, 2017.8 tps, lat 49.870 ms stddev 59.030 2024-09-16T01:14:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:11 smithi193 ceph-mon[35701]: pgmap v104: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-09-16T01:14:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:11 smithi148 ceph-mon[35030]: pgmap v104: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-09-16T01:14:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:11 smithi078 ceph-mon[30937]: pgmap v104: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 1.82k op/s 2024-09-16T01:14:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:13 smithi148 ceph-mon[35030]: pgmap v105: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-09-16T01:14:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:13 smithi078 ceph-mon[30937]: pgmap v105: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-09-16T01:14:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:13 smithi193 ceph-mon[35701]: pgmap v105: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-09-16T01:14:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:15 smithi148 ceph-mon[35030]: pgmap v106: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-16T01:14:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:15 smithi078 ceph-mon[30937]: pgmap v106: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-16T01:14:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:15 smithi193 ceph-mon[35701]: pgmap v106: 129 pgs: 129 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-16T01:14:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 180.0 s, 1976.6 tps, lat 50.547 ms stddev 56.755 2024-09-16T01:14:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:17 smithi148 ceph-mon[35030]: pgmap v107: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 22 MiB/s wr, 1.72k op/s 2024-09-16T01:14:17.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:17 smithi078 ceph-mon[30937]: pgmap v107: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 22 MiB/s wr, 1.72k op/s 2024-09-16T01:14:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:17 smithi193 ceph-mon[35701]: pgmap v107: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 22 MiB/s wr, 1.72k op/s 2024-09-16T01:14:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:19 smithi148 ceph-mon[35030]: pgmap v108: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 953 op/s 2024-09-16T01:14:19.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:19 smithi078 ceph-mon[30937]: pgmap v108: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 953 op/s 2024-09-16T01:14:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:19 smithi193 ceph-mon[35701]: pgmap v108: 129 pgs: 129 active+clean; 1.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 953 op/s 2024-09-16T01:14:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 185.0 s, 1685.6 tps, lat 59.194 ms stddev 63.315 2024-09-16T01:14:21.348 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:21 smithi193 ceph-mon[35701]: pgmap v109: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-09-16T01:14:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:21 smithi148 ceph-mon[35030]: pgmap v109: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-09-16T01:14:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:21 smithi078 ceph-mon[30937]: pgmap v109: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.848+0000 7f19c7182640 1 -- 172.21.15.78:0/1775467285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0112900 msgr2=0x7f19c0114cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.848+0000 7f19c7182640 1 --2- 172.21.15.78:0/1775467285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0112900 0x7f19c0114cf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009ca0 tx=0x7f19b802a970 comp rx=0 tx=0).stop 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.849+0000 7f19c7182640 1 -- 172.21.15.78:0/1775467285 shutdown_connections 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.849+0000 7f19c7182640 1 --2- 172.21.15.78:0/1775467285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0112900 0x7f19c0114cf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.849+0000 7f19c7182640 1 --2- 172.21.15.78:0/1775467285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c0074eb0 0x7f19c01123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.849+0000 7f19c7182640 1 --2- 172.21.15.78:0/1775467285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0074590 0x7f19c0074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.849+0000 7f19c7182640 1 -- 172.21.15.78:0/1775467285 >> 172.21.15.78:0/1775467285 conn(0x7f19c00fd5f0 msgr2=0x7f19c00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.851+0000 7f19c7182640 1 -- 172.21.15.78:0/1775467285 shutdown_connections 2024-09-16T01:14:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.853+0000 7f19c7182640 1 -- 172.21.15.78:0/1775467285 wait complete. 2024-09-16T01:14:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.854+0000 7f19c7182640 1 Processor -- start 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 -- start start 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c0074590 0x7f19c01a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0112900 0x7f19c019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0118410 con 0x7f19c0074590 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0118290 con 0x7f19c0074eb0 2024-09-16T01:14:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c7182640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0118590 con 0x7f19c0112900 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c597f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c597f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50310/0 (socket says 172.21.15.78:50310) 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c597f640 1 -- 172.21.15.78:0/1326086921 learned_addr learned my addr 172.21.15.78:0/1326086921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.855+0000 7f19c6981640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0112900 0x7f19c019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 -- 172.21.15.78:0/1326086921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0112900 msgr2=0x7f19c019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0112900 0x7f19c019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 -- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c0074590 msgr2=0x7f19c01a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c0074590 0x7f19c01a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 -- 172.21.15.78:0/1326086921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c019ee80 con 0x7f19c0074eb0 2024-09-16T01:14:21.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.856+0000 7f19c597f640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00b4d0 tx=0x7f19bc00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.859+0000 7f19b77fe640 1 -- 172.21.15.78:0/1326086921 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc014070 con 0x7f19c0074eb0 2024-09-16T01:14:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.859+0000 7f19b77fe640 1 -- 172.21.15.78:0/1326086921 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19bc010d50 con 0x7f19c0074eb0 2024-09-16T01:14:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.860+0000 7f19b77fe640 1 -- 172.21.15.78:0/1326086921 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc017360 con 0x7f19c0074eb0 2024-09-16T01:14:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.860+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c019f170 con 0x7f19c0074eb0 2024-09-16T01:14:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.860+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c0105b90 con 0x7f19c0074eb0 2024-09-16T01:14:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.864+0000 7f19cb25b740 1 Processor -- start 2024-09-16T01:14:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.866+0000 7f19b77fe640 1 -- 172.21.15.78:0/1326086921 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f19bc017070 con 0x7f19c0074eb0 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 -- start start 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559af6a402f0 con 0x559af6adbcc0 2024-09-16T01:14:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559af69e8270 con 0x559af6adc5e0 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19cb25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559af6a3c2b0 con 0x559af6ae6e80 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19c6981640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.869+0000 7f19c6180640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33650/0 (socket says 172.21.15.78:33650) 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 -- 172.21.15.78:0/2418167645 learned_addr learned my addr 172.21.15.78:0/2418167645 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c597f640 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 -- 172.21.15.78:0/2418167645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 msgr2=0x559af6ae9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 -- 172.21.15.78:0/2418167645 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 msgr2=0x559af6ae6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.870+0000 7f19c6981640 1 -- 172.21.15.78:0/2418167645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af69e7340 con 0x559af6adbcc0 2024-09-16T01:14:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.871+0000 7f19b77fe640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 0x7f19ac0a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.871+0000 7f19b77fe640 1 -- 172.21.15.78:0/1326086921 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f19bc0cf640 con 0x7f19c0074eb0 2024-09-16T01:14:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.871+0000 7f19c6981640 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f19b802af60 tx=0x7f19b802ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5df87707500f68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.872+0000 7f19c6180640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 0x7f19ac0a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.872+0000 7f19b4bfe640 1 -- 172.21.15.78:0/2418167645 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8049070 con 0x559af6adbcc0 2024-09-16T01:14:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.872+0000 7f19b4bfe640 1 -- 172.21.15.78:0/2418167645 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19b8003e50 con 0x559af6adbcc0 2024-09-16T01:14:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19b4bfe640 1 -- 172.21.15.78:0/2418167645 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8034070 con 0x559af6adbcc0 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 -- 172.21.15.78:0/2418167645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 msgr2=0x559af6adc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f19b802af60 tx=0x7f19b802ae70 comp rx=0 tx=0).stop 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19c6180640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 0x7f19ac0a58e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x559af6ad9090 tx=0x7f19b00090e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 -- 172.21.15.78:0/2418167645 shutdown_connections 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 --2- 172.21.15.78:0/2418167645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6adc0a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 -- 172.21.15.78:0/2418167645 >> 172.21.15.78:0/2418167645 conn(0x559af6ad8230 msgr2=0x559af6ad8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 -- 172.21.15.78:0/2418167645 shutdown_connections 2024-09-16T01:14:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.873+0000 7f19cb25b740 1 -- 172.21.15.78:0/2418167645 wait complete. 2024-09-16T01:14:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 -- start start 2024-09-16T01:14:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 Processor -- start 2024-09-16T01:14:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ae3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559af6ae9aa0 con 0x559af6adbcc0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559af69c9290 con 0x559af6adc5e0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559af6ae9c20 con 0x559af6ae6e80 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19cb25b740 10 client.0 _ll_register_callbacks cb 0x559af69fa140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19c597f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19c597f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50328/0 (socket says 172.21.15.78:50328) 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19c597f640 1 -- 172.21.15.78:0/4221472907 learned_addr learned my addr 172.21.15.78:0/4221472907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19c6180640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ae3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.874+0000 7f19c6981640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 -- 172.21.15.78:0/4221472907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 msgr2=0x559af6ae4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 -- 172.21.15.78:0/4221472907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 msgr2=0x559af6ae3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ae3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 -- 172.21.15.78:0/4221472907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af6753210 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19c597f640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00be90 tx=0x7f19bc097940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb9963e038a8d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc09b000 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b8003e50 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f19bc0afbc0 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19bc00fed0 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.875+0000 7f19cb25b740 10 client.25280 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559af681f620 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19b8003e50 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f19bc09aaf0 con 0x559af6adc5e0 2024-09-16T01:14:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 10 client.25280 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 10 client.25280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 20 client.25280 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6af6e90 0x559af6af92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 4 client.25280 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:14:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.876+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559af6a4da20 con 0x559af6af6e90 2024-09-16T01:14:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.878+0000 7f19c6180640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6af6e90 0x559af6af92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19a9ffb640 10 client.25280 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19c6180640 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6af6e90 0x559af6af92a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38c3198a3e77ff0f server_cookie=28d0652770012115 in_seq=0 out_seq=0 2024-09-16T01:14:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19a9ffb640 1 -- 172.21.15.78:0/4221472907 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559af6a4da20 con 0x559af6af6e90 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19a9ffb640 10 client.25280 handle_command_reply: tid=0 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19cb25b740 1 client.25280 shutdown 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 msgr2=0x559af6ae42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.879+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00be90 tx=0x7f19bc097940 comp rx=0 tx=0).stop 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 shutdown_connections 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6af6e90 0x559af6af92a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6ae4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19a9ffb640 0 client.25280 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae42c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 --2- 172.21.15.78:0/4221472907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ae3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 >> 172.21.15.78:0/4221472907 conn(0x559af6ad8230 msgr2=0x559af6addf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 shutdown_connections 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 1 -- 172.21.15.78:0/4221472907 wait complete. 2024-09-16T01:14:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.880+0000 7f19cb25b740 20 client.25280 trim_cache size 0 max 16384 2024-09-16T01:14:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.903+0000 7f19cb25b740 1 Processor -- start 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.903+0000 7f19cb25b740 1 -- start start 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.903+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ade780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.903+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6afeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19cb25b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01c0190 con 0x559af6adbcc0 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19cb25b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01bda60 con 0x559af6adc5e0 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19cb25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0075cd0 con 0x559af6ae6e80 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c6180640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ade780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c6180640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ade780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33660/0 (socket says 172.21.15.78:33660) 2024-09-16T01:14:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c6180640 1 -- 172.21.15.78:0/3789283372 learned_addr learned my addr 172.21.15.78:0/3789283372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c6981640 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6afeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 -- 172.21.15.78:0/3789283372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 msgr2=0x559af6afeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6afeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 -- 172.21.15.78:0/3789283372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 msgr2=0x559af6ade780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ade780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 -- 172.21.15.78:0/3789283372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af6ad8ec0 con 0x559af6adc5e0 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19c597f640 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae4630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ccbc0 tx=0x7f19bc097a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a002af19a08a977 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.904+0000 7f19abfff640 1 -- 172.21.15.78:0/3789283372 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0cf490 con 0x559af6adc5e0 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.905+0000 7f19abfff640 1 -- 172.21.15.78:0/3789283372 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19bc0cfd50 con 0x559af6adc5e0 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.905+0000 7f19abfff640 1 -- 172.21.15.78:0/3789283372 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0973f0 con 0x559af6adc5e0 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.906+0000 7f19cb25b740 1 -- 172.21.15.78:0/3789283372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 msgr2=0x559af6ae4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.906+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae4630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0ccbc0 tx=0x7f19bc097a70 comp rx=0 tx=0).stop 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 -- 172.21.15.78:0/3789283372 shutdown_connections 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6ae6e80 0x559af6afeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6adc5e0 0x559af6ae4630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3789283372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6ade780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 -- 172.21.15.78:0/3789283372 >> 172.21.15.78:0/3789283372 conn(0x559af6ad8230 msgr2=0x559af6ae5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 -- 172.21.15.78:0/3789283372 shutdown_connections 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 -- 172.21.15.78:0/3789283372 wait complete. 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 -- start start 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 Processor -- start 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.907+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6aef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6ae6e80 0x559af6ae13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f19c010e0e0 con 0x559af6adbcc0 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01bd390 con 0x559af6ae6e80 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01bc5f0 con 0x559af6adc5e0 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19cb25b740 10 client.0 _ll_register_callbacks cb 0x7f19bc097280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:21.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c6180640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6aef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c6180640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6aef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33672/0 (socket says 172.21.15.78:33672) 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:39074/0 (socket says 172.21.15.78:39074) 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c6180640 1 -- 172.21.15.78:0/3739284834 learned_addr learned my addr 172.21.15.78:0/3739284834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c6981640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6ae6e80 0x559af6ae13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 -- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6ae6e80 msgr2=0x559af6ae13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6ae6e80 0x559af6ae13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 -- 172.21.15.78:0/3739284834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 msgr2=0x559af6aef7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6aef7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 -- 172.21.15.78:0/3739284834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af6b60b40 con 0x559af6adc5e0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19c597f640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f19bc09b170 tx=0x7f19bc097940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722fe76985459e5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.908+0000 7f19a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:14:21.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19bc09bda0 con 0x559af6adc5e0 2024-09-16T01:14:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f19bc0af8a0 con 0x559af6adc5e0 2024-09-16T01:14:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559af6a47d70 con 0x559af6adc5e0 2024-09-16T01:14:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19bc09b6a0 con 0x559af6adc5e0 2024-09-16T01:14:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19cb25b740 10 client.25336 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.909+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559af6b61d30 con 0x559af6adc5e0 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19bc09fa50 con 0x559af6adc5e0 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f19bc097550 con 0x559af6adc5e0 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 10 client.25336 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 10 client.25336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 20 client.25336 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6b62320 0x559af6b64740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 4 client.25336 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559af6a47d70 con 0x559af6b62320 2024-09-16T01:14:21.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.911+0000 7f19c6180640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6b62320 0x559af6b64740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.912+0000 7f19a9ffb640 10 client.25336 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:21.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.912+0000 7f19c6180640 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6b62320 0x559af6b64740 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c561e6a82760b79 server_cookie=3b67523f8af50516 in_seq=0 out_seq=0 2024-09-16T01:14:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19a9ffb640 1 -- 172.21.15.78:0/3739284834 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559af6a47d70 con 0x559af6b62320 2024-09-16T01:14:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19a9ffb640 10 client.25336 handle_command_reply: tid=0 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 client.25336 shutdown 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 msgr2=0x559af6ae0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f19bc09b170 tx=0x7f19bc097940 comp rx=0 tx=0).stop 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 shutdown_connections 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559af6b62320 0x559af6b64740 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559af6ae6e80 0x559af6ae13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19a9ffb640 0 client.25336 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559af6adc5e0 0x559af6ae0c70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 --2- 172.21.15.78:0/3739284834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559af6adbcc0 0x559af6aef7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 >> 172.21.15.78:0/3739284834 conn(0x559af6ad8230 msgr2=0x559af6ad7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 shutdown_connections 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 1 -- 172.21.15.78:0/3739284834 wait complete. 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.919+0000 7f19cb25b740 20 client.25336 trim_cache size 0 max 16384 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:14:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 msgr2=0x7f19ac0a58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 0x7f19ac0a58e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x559af6ad9090 tx=0x7f19b00090e0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 msgr2=0x7f19c01a4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00b4d0 tx=0x7f19bc00b990 comp rx=0 tx=0).stop 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 shutdown_connections 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19ac0a3420 0x7f19ac0a58e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19c0112900 0x7f19c019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19c0074eb0 0x7f19c01a4af0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 --2- 172.21.15.78:0/1326086921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c0074590 0x7f19c01a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.920+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 >> 172.21.15.78:0/1326086921 conn(0x7f19c00fd5f0 msgr2=0x7f19c00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.921+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 shutdown_connections 2024-09-16T01:14:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:21.921+0000 7f19c7182640 1 -- 172.21.15.78:0/1326086921 wait complete. 2024-09-16T01:14:23.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:23 smithi148 ceph-mon[35030]: pgmap v110: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-09-16T01:14:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:23 smithi078 ceph-mon[30937]: pgmap v110: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-09-16T01:14:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:23 smithi193 ceph-mon[35701]: pgmap v110: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-09-16T01:14:25.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:25 smithi148 ceph-mon[35030]: pgmap v111: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-09-16T01:14:25.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:25 smithi078 ceph-mon[30937]: pgmap v111: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-09-16T01:14:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:25 smithi193 ceph-mon[35701]: pgmap v111: 129 pgs: 129 active+clean; 1.9 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-09-16T01:14:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 190.0 s, 1636.4 tps, lat 60.882 ms stddev 70.457 2024-09-16T01:14:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:27 smithi148 ceph-mon[35030]: pgmap v112: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.4 MiB/s wr, 827 op/s 2024-09-16T01:14:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:27 smithi078 ceph-mon[30937]: pgmap v112: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.4 MiB/s wr, 827 op/s 2024-09-16T01:14:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:27 smithi193 ceph-mon[35701]: pgmap v112: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.4 MiB/s wr, 827 op/s 2024-09-16T01:14:29.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:29 smithi148 ceph-mon[35030]: pgmap v113: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.9 MiB/s wr, 561 op/s 2024-09-16T01:14:29.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:29 smithi078 ceph-mon[30937]: pgmap v113: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.9 MiB/s wr, 561 op/s 2024-09-16T01:14:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:29 smithi193 ceph-mon[35701]: pgmap v113: 129 pgs: 129 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.9 MiB/s wr, 561 op/s 2024-09-16T01:14:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 195.0 s, 2058.8 tps, lat 48.669 ms stddev 54.231 2024-09-16T01:14:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:31 smithi148 ceph-mon[35030]: pgmap v114: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-09-16T01:14:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:31 smithi078 ceph-mon[30937]: pgmap v114: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-09-16T01:14:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:31 smithi193 ceph-mon[35701]: pgmap v114: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.5 MiB/s wr, 844 op/s 2024-09-16T01:14:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:33 smithi148 ceph-mon[35030]: pgmap v115: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 578 op/s 2024-09-16T01:14:33.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:33 smithi078 ceph-mon[30937]: pgmap v115: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 578 op/s 2024-09-16T01:14:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:33 smithi193 ceph-mon[35701]: pgmap v115: 129 pgs: 129 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 578 op/s 2024-09-16T01:14:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f49e1640 1 -- 172.21.15.78:0/1732905221 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0319b0 con 0x7f72f01069e0 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 -- 172.21.15.78:0/1732905221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 msgr2=0x7f72f0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 --2- 172.21.15.78:0/1732905221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0106dc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009930 tx=0x7f72ec02a460 comp rx=0 tx=0).stop 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 -- 172.21.15.78:0/1732905221 shutdown_connections 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 --2- 172.21.15.78:0/1732905221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f0101370 0x7f72f010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 --2- 172.21.15.78:0/1732905221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f01009d0 0x7f72f0100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 --2- 172.21.15.78:0/1732905221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0106dc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 -- 172.21.15.78:0/1732905221 >> 172.21.15.78:0/1732905221 conn(0x7f72f00fc6b0 msgr2=0x7f72f00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 -- 172.21.15.78:0/1732905221 shutdown_connections 2024-09-16T01:14:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.050+0000 7f72f7c47640 1 -- 172.21.15.78:0/1732905221 wait complete. 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.051+0000 7f72f7c47640 1 Processor -- start 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 -- start start 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 0x7f72f0107e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0108380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0110ef0 con 0x7f72f0101370 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0110d70 con 0x7f72f01009d0 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f7c47640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0111070 con 0x7f72f01069e0 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33232/0 (socket says 172.21.15.78:33232) 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f59e3640 1 -- 172.21.15.78:0/3931083179 learned_addr learned my addr 172.21.15.78:0/3931083179 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f61e4640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0108380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.052+0000 7f72f51e2640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 0x7f72f0107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 -- 172.21.15.78:0/3931083179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 msgr2=0x7f72f0108380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0108380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 -- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 msgr2=0x7f72f0107e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 0x7f72f0107e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 -- 172.21.15.78:0/3931083179 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f019a4d0 con 0x7f72f01009d0 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f61e4640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f51e2640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 0x7f72f0107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f59e3640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72ec02a980 tx=0x7f72ec003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72e6ffd640 1 -- 172.21.15.78:0/3931083179 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec03f070 con 0x7f72f01009d0 2024-09-16T01:14:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72e6ffd640 1 -- 172.21.15.78:0/3931083179 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ec03b460 con 0x7f72f01009d0 2024-09-16T01:14:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f019a760 con 0x7f72f01009d0 2024-09-16T01:14:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72e6ffd640 1 -- 172.21.15.78:0/3931083179 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec035030 con 0x7f72f01009d0 2024-09-16T01:14:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.053+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f019aca0 con 0x7f72f01009d0 2024-09-16T01:14:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.057+0000 7f72e6ffd640 1 -- 172.21.15.78:0/3931083179 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f72ec042070 con 0x7f72f01009d0 2024-09-16T01:14:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 Processor -- start 2024-09-16T01:14:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 -- start start 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5612c341c2f0 con 0x5612c34b7af0 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5612c33c4270 con 0x5612c34c2cb0 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.058+0000 7f72fa9db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5612c34182b0 con 0x5612c34b8410 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.059+0000 7f72f51e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.059+0000 7f72f51e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33240/0 (socket says 172.21.15.78:33240) 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.059+0000 7f72f51e2640 1 -- 172.21.15.78:0/1067741475 learned_addr learned my addr 172.21.15.78:0/1067741475 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.059+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.059+0000 7f72f61e4640 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 -- 172.21.15.78:0/1067741475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 msgr2=0x5612c34c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34c2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 -- 172.21.15.78:0/1067741475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 msgr2=0x5612c34b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 -- 172.21.15.78:0/1067741475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c33c3340 con 0x5612c34c2cb0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72e6ffd640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 0x7f72dc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.060+0000 7f72f51e2640 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72e00077b0 tx=0x7f72e0007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38065c0772ef2d6f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.061+0000 7f72e6ffd640 1 -- 172.21.15.78:0/3931083179 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f72ec0f0ac0 con 0x7f72f01009d0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.061+0000 7f72f61e4640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 0x7f72dc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.061+0000 7f72dbfff640 1 -- 172.21.15.78:0/1067741475 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e000d950 con 0x5612c34c2cb0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.061+0000 7f72dbfff640 1 -- 172.21.15.78:0/1067741475 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72e0018070 con 0x5612c34c2cb0 2024-09-16T01:14:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.061+0000 7f72dbfff640 1 -- 172.21.15.78:0/1067741475 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e00123e0 con 0x5612c34c2cb0 2024-09-16T01:14:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.062+0000 7f72fa9db740 1 -- 172.21.15.78:0/1067741475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 msgr2=0x5612c34c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.062+0000 7f72f61e4640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 0x7f72dc0a59a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72e800a870 tx=0x7f72e800d040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.062+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72e00077b0 tx=0x7f72e0007c70 comp rx=0 tx=0).stop 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 -- 172.21.15.78:0/1067741475 shutdown_connections 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c2cb0 0x5612c34c50a0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f72e00077b0 tx=0x7f72e0007c70 comp rx=0 tx=0).stop 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34c2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1067741475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 -- 172.21.15.78:0/1067741475 >> 172.21.15.78:0/1067741475 conn(0x5612c34b4060 msgr2=0x5612c34b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.063+0000 7f72fa9db740 1 -- 172.21.15.78:0/1067741475 shutdown_connections 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.064+0000 7f72fa9db740 1 -- 172.21.15.78:0/1067741475 wait complete. 2024-09-16T01:14:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.064+0000 7f72fa9db740 1 -- start start 2024-09-16T01:14:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.064+0000 7f72fa9db740 1 Processor -- start 2024-09-16T01:14:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34c02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5612c34c58d0 con 0x5612c34b8410 2024-09-16T01:14:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5612c33a5290 con 0x5612c34b7af0 2024-09-16T01:14:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5612c34c5a50 con 0x5612c34c0800 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33248/0 (socket says 172.21.15.78:33248) 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72f59e3640 1 -- 172.21.15.78:0/661087893 learned_addr learned my addr 172.21.15.78:0/661087893 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72f61e4640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.065+0000 7f72fa9db740 10 client.0 _ll_register_callbacks cb 0x5612c33d6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f59e3640 1 -- 172.21.15.78:0/661087893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 msgr2=0x5612c34cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f59e3640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f59e3640 1 -- 172.21.15.78:0/661087893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 msgr2=0x5612c34c02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f59e3640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34c02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f59e3640 1 -- 172.21.15.78:0/661087893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c312f210 con 0x5612c34b7af0 2024-09-16T01:14:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.066+0000 7f72f61e4640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.067+0000 7f72f59e3640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0ba320 tx=0x7f72ec009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9fa704669dc5a6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.067+0000 7f72d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.067+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec049070 con 0x5612c34b7af0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.067+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f72ec0b9fc0 con 0x5612c34b7af0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.067+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72e0018070 con 0x5612c34b7af0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.068+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f72ec0bd490 con 0x5612c34b7af0 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.068+0000 7f72fa9db740 10 client.25295 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.068+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612c31fb620 con 0x5612c34b7af0 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.068+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72e0018070 con 0x5612c34b7af0 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f72ec03b610 con 0x5612c34b7af0 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 10 client.25295 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 10 client.25295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 20 client.25295 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c34d56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 4 client.25295 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612c3429a20 con 0x5612c34d3230 2024-09-16T01:14:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.069+0000 7f72f51e2640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c34d56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.070+0000 7f72f51e2640 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c34d56c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a262f30d96b2fa server_cookie=eed64a4531f68e11 in_seq=0 out_seq=0 2024-09-16T01:14:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.071+0000 7f72d97fa640 10 client.25295 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.071+0000 7f72d97fa640 1 -- 172.21.15.78:0/661087893 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612c3429a20 con 0x5612c34d3230 2024-09-16T01:14:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.071+0000 7f72d97fa640 10 client.25295 handle_command_reply: tid=0 2024-09-16T01:14:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.071+0000 7f72fa9db740 1 client.25295 shutdown 2024-09-16T01:14:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 msgr2=0x5612c34bfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0ba320 tx=0x7f72ec009620 comp rx=0 tx=0).stop 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 shutdown_connections 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c34d56c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72d97fa640 0 client.25295 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 --2- 172.21.15.78:0/661087893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34bfd80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 >> 172.21.15.78:0/661087893 conn(0x5612c34b4060 msgr2=0x5612c34b9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 shutdown_connections 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 1 -- 172.21.15.78:0/661087893 wait complete. 2024-09-16T01:14:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.072+0000 7f72fa9db740 20 client.25295 trim_cache size 0 max 16384 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.096+0000 7f72fa9db740 1 Processor -- start 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 -- start start 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34db510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01bffe0 con 0x5612c34b8410 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01c06b0 con 0x5612c34b7af0 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.097+0000 7f72fa9db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01bdf80 con 0x5612c34c0800 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.098+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.098+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33256/0 (socket says 172.21.15.78:33256) 2024-09-16T01:14:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.098+0000 7f72f59e3640 1 -- 172.21.15.78:0/1245115557 learned_addr learned my addr 172.21.15.78:0/1245115557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f61e4640 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34db510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f59e3640 1 -- 172.21.15.78:0/1245115557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 msgr2=0x5612c34db510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34db510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f59e3640 1 -- 172.21.15.78:0/1245115557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 msgr2=0x5612c34dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.102+0000 7f72f59e3640 1 -- 172.21.15.78:0/1245115557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c34c0630 con 0x5612c34b7af0 2024-09-16T01:14:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.103+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0b9db0 tx=0x7f72ec009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cc9eebd67c8489 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.109+0000 7f72db7fe640 1 -- 172.21.15.78:0/1245115557 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec049070 con 0x5612c34b7af0 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.109+0000 7f72db7fe640 1 -- 172.21.15.78:0/1245115557 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ec0f8070 con 0x5612c34b7af0 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.110+0000 7f72fa9db740 1 -- 172.21.15.78:0/1245115557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 msgr2=0x5612c34ceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.110+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0b9db0 tx=0x7f72ec009260 comp rx=0 tx=0).stop 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.111+0000 7f72fa9db740 1 -- 172.21.15.78:0/1245115557 shutdown_connections 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.111+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34c0800 0x5612c34db510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.111+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b8410 0x5612c34dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.111+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1245115557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34b7af0 0x5612c34ceba0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.111+0000 7f72fa9db740 1 -- 172.21.15.78:0/1245115557 >> 172.21.15.78:0/1245115557 conn(0x5612c34b4060 msgr2=0x5612c34bb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.115+0000 7f72fa9db740 1 -- 172.21.15.78:0/1245115557 shutdown_connections 2024-09-16T01:14:35.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.119+0000 7f72fa9db740 1 -- 172.21.15.78:0/1245115557 wait complete. 2024-09-16T01:14:35.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.119+0000 7f72fa9db740 1 -- start start 2024-09-16T01:14:35.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.119+0000 7f72fa9db740 1 Processor -- start 2024-09-16T01:14:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c0800 0x5612c34bc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01a2a50 con 0x5612c34b7af0 2024-09-16T01:14:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01bd8b0 con 0x5612c34c0800 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01bcb10 con 0x5612c34b8410 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72fa9db740 10 client.0 _ll_register_callbacks cb 0x7f72ec0f41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72f59e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57358/0 (socket says 172.21.15.78:57358) 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.123+0000 7f72f59e3640 1 -- 172.21.15.78:0/1801753060 learned_addr learned my addr 172.21.15.78:0/1801753060 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.124+0000 7f72f59e3640 1 -- 172.21.15.78:0/1801753060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 msgr2=0x5612c34cb250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.124+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34cb250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.124+0000 7f72f59e3640 1 -- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c0800 msgr2=0x5612c34bc3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:14:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.124+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c0800 0x5612c34bc3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.124+0000 7f72f59e3640 1 -- 172.21.15.78:0/1801753060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c34bc900 con 0x5612c34b7af0 2024-09-16T01:14:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.125+0000 7f72f59e3640 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009900 tx=0x7f72ec0bbe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dc935091271b9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:35.126 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.125+0000 7f72d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:14:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.128+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec049050 con 0x5612c34b7af0 2024-09-16T01:14:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.128+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ec0f8070 con 0x5612c34b7af0 2024-09-16T01:14:35.129 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.128+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f72ec0bcda0 con 0x5612c34b7af0 2024-09-16T01:14:35.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.131+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f72ec0be050 con 0x5612c34b7af0 2024-09-16T01:14:35.136 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.136+0000 7f72fa9db740 10 client.15840 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:35.137 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.136+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612c34bc900 con 0x5612c34b7af0 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ec0f8070 con 0x5612c34b7af0 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f72ec0b9800 con 0x5612c34b7af0 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 10 client.15840 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 20 client.15840 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c3550b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 4 client.15840 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:14:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.140+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612c3423d70 con 0x5612c34d3230 2024-09-16T01:14:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.141+0000 7f72f51e2640 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c3550b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.153+0000 7f72f51e2640 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c3550b60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=642ea28252a52e95 server_cookie=2d47f7c16a96542e in_seq=0 out_seq=0 2024-09-16T01:14:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.153+0000 7f72d97fa640 10 client.15840 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72d97fa640 1 -- 172.21.15.78:0/1801753060 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612c3423d70 con 0x5612c34d3230 2024-09-16T01:14:35.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72d97fa640 10 client.15840 handle_command_reply: tid=0 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 client.15840 shutdown 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 msgr2=0x5612c34cad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009900 tx=0x7f72ec0bbe60 comp rx=0 tx=0).stop 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 shutdown_connections 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5612c34d3230 0x5612c3550b60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612c34c0800 0x5612c34bc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5612c34b8410 0x5612c34cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72d97fa640 0 client.15840 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 --2- 172.21.15.78:0/1801753060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612c34b7af0 0x5612c34cad10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 >> 172.21.15.78:0/1801753060 conn(0x5612c34b4060 msgr2=0x5612c34b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 shutdown_connections 2024-09-16T01:14:35.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 1 -- 172.21.15.78:0/1801753060 wait complete. 2024-09-16T01:14:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.194+0000 7f72fa9db740 20 client.15840 trim_cache size 0 max 16384 2024-09-16T01:14:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:14:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:14:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:14:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 msgr2=0x7f72dc0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 0x7f72dc0a59a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f72e800a870 tx=0x7f72e800d040 comp rx=0 tx=0).stop 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 msgr2=0x7f72f0107900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f72ec02a980 tx=0x7f72ec003c80 comp rx=0 tx=0).stop 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 shutdown_connections 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f72dc0a34e0 0x7f72dc0a59a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f01069e0 0x7f72f0108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f0101370 0x7f72f0107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 --2- 172.21.15.78:0/3931083179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72f01009d0 0x7f72f0107900 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.195+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 >> 172.21.15.78:0/3931083179 conn(0x7f72f00fc6b0 msgr2=0x7f72f006a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.196+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 shutdown_connections 2024-09-16T01:14:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:35.196+0000 7f72f7c47640 1 -- 172.21.15.78:0/3931083179 wait complete. 2024-09-16T01:14:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:35 smithi148 ceph-mon[35030]: pgmap v116: 129 pgs: 129 active+clean; 1.9 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 620 op/s 2024-09-16T01:14:35.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:35 smithi078 ceph-mon[30937]: pgmap v116: 129 pgs: 129 active+clean; 1.9 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 620 op/s 2024-09-16T01:14:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:35 smithi193 ceph-mon[35701]: pgmap v116: 129 pgs: 129 active+clean; 1.9 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.6 MiB/s wr, 620 op/s 2024-09-16T01:14:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 200.0 s, 2027.0 tps, lat 49.220 ms stddev 72.872 2024-09-16T01:14:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:37 smithi148 ceph-mon[35030]: pgmap v117: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:14:37.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:37 smithi078 ceph-mon[30937]: pgmap v117: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:14:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:37 smithi193 ceph-mon[35701]: pgmap v117: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:14:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:39 smithi148 ceph-mon[35030]: pgmap v118: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-09-16T01:14:39.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:39 smithi078 ceph-mon[30937]: pgmap v118: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-09-16T01:14:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:39 smithi193 ceph-mon[35701]: pgmap v118: 129 pgs: 129 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 941 op/s 2024-09-16T01:14:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 205.0 s, 1353.8 tps, lat 74.005 ms stddev 80.170 2024-09-16T01:14:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:41 smithi148 ceph-mon[35030]: pgmap v119: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-09-16T01:14:41.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:41 smithi078 ceph-mon[30937]: pgmap v119: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-09-16T01:14:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:41 smithi193 ceph-mon[35701]: pgmap v119: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-09-16T01:14:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:43 smithi148 ceph-mon[35030]: pgmap v120: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 10 MiB/s wr, 927 op/s 2024-09-16T01:14:43.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:43 smithi078 ceph-mon[30937]: pgmap v120: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 10 MiB/s wr, 927 op/s 2024-09-16T01:14:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:43 smithi193 ceph-mon[35701]: pgmap v120: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 10 MiB/s wr, 927 op/s 2024-09-16T01:14:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:45 smithi148 ceph-mon[35030]: pgmap v121: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 962 op/s 2024-09-16T01:14:45.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:45 smithi078 ceph-mon[30937]: pgmap v121: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 962 op/s 2024-09-16T01:14:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:45 smithi193 ceph-mon[35701]: pgmap v121: 129 pgs: 129 active+clean; 2.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 962 op/s 2024-09-16T01:14:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 210.0 s, 2402.2 tps, lat 41.223 ms stddev 52.921 2024-09-16T01:14:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:47 smithi148 ceph-mon[35030]: pgmap v122: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-09-16T01:14:47.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:47 smithi078 ceph-mon[30937]: pgmap v122: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-09-16T01:14:47.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:47 smithi193 ceph-mon[35701]: pgmap v122: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-09-16T01:14:48.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 -- 172.21.15.78:0/4148146430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c107ad0 msgr2=0x7fe81c107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 --2- 172.21.15.78:0/4148146430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c107ad0 0x7fe81c107eb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009cd0 tx=0x7fe80c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 -- 172.21.15.78:0/4148146430 shutdown_connections 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 --2- 172.21.15.78:0/4148146430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c102460 0x7fe81c10e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 --2- 172.21.15.78:0/4148146430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c101ac0 0x7fe81c101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 --2- 172.21.15.78:0/4148146430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c107ad0 0x7fe81c107eb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 -- 172.21.15.78:0/4148146430 >> 172.21.15.78:0/4148146430 conn(0x7fe81c0fd820 msgr2=0x7fe81c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 -- 172.21.15.78:0/4148146430 shutdown_connections 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.331+0000 7fe8217a8640 1 -- 172.21.15.78:0/4148146430 wait complete. 2024-09-16T01:14:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 Processor -- start 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 -- start start 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c102460 0x7fe81c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 0x7fe81c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c111f10 con 0x7fe81c101ac0 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c111d90 con 0x7fe81c107ad0 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe8217a8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c112090 con 0x7fe81c102460 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe81affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe81affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39904/0 (socket says 172.21.15.78:39904) 2024-09-16T01:14:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe81affd640 1 -- 172.21.15.78:0/3463165867 learned_addr learned my addr 172.21.15.78:0/3463165867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.332+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 0x7fe81c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c102460 0x7fe81c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81b7fe640 1 -- 172.21.15.78:0/3463165867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c102460 msgr2=0x7fe81c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c102460 0x7fe81c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81b7fe640 1 -- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 msgr2=0x7fe81c198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81b7fe640 1 -- 172.21.15.78:0/3463165867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c19d370 con 0x7fe81c107ad0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.333+0000 7fe81affd640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.334+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 0x7fe81c19cc40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.334+0000 7fe803fff640 1 -- 172.21.15.78:0/3463165867 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81400ccc0 con 0x7fe81c107ad0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.334+0000 7fe803fff640 1 -- 172.21.15.78:0/3463165867 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe814005020 con 0x7fe81c107ad0 2024-09-16T01:14:48.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.335+0000 7fe803fff640 1 -- 172.21.15.78:0/3463165867 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814004a00 con 0x7fe81c107ad0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.335+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c19d660 con 0x7fe81c107ad0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.335+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1a4ef0 con 0x7fe81c107ad0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 Processor -- start 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe803fff640 1 -- 172.21.15.78:0/3463165867 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fe8140051d0 con 0x7fe81c107ad0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 -- start start 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638d0410 0x55dd638da770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638dacb0 0x55dd638dd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd638342f0 con 0x55dd638cfaf0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dd637dc270 con 0x55dd638dacb0 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe82461d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dd638302b0 con 0x55dd638d0410 2024-09-16T01:14:48.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39916/0 (socket says 172.21.15.78:39916) 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe81a7fc640 1 -- 172.21.15.78:0/1987215409 learned_addr learned my addr 172.21.15.78:0/1987215409 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638dacb0 0x55dd638dd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.338+0000 7fe81affd640 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638d0410 0x55dd638da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe803fff640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 0x7fe8040a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 -- 172.21.15.78:0/1987215409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638d0410 msgr2=0x55dd638da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638d0410 0x55dd638da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 -- 172.21.15.78:0/1987215409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638dacb0 msgr2=0x55dd638dd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638dacb0 0x55dd638dd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 -- 172.21.15.78:0/1987215409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd637db340 con 0x55dd638cfaf0 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 0x7fe8040a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe803fff640 1 -- 172.21.15.78:0/3463165867 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe8140d0020 con 0x7fe81c107ad0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.339+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe810013500 tx=0x7fe81000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3f2ac47b26a4a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.340+0000 7fe8017fa640 1 -- 172.21.15.78:0/1987215409 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810005020 con 0x55dd638cfaf0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.340+0000 7fe8017fa640 1 -- 172.21.15.78:0/1987215409 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8100047a0 con 0x55dd638cfaf0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.340+0000 7fe8017fa640 1 -- 172.21.15.78:0/1987215409 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8100119e0 con 0x55dd638cfaf0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 -- 172.21.15.78:0/1987215409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 msgr2=0x55dd638cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe81b7fe640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 0x7fe8040a5a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe8140cd490 tx=0x7fe8140c6d40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe810013500 tx=0x7fe81000b6a0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 -- 172.21.15.78:0/1987215409 shutdown_connections 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638dacb0 0x55dd638dd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638d0410 0x55dd638da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 --2- 172.21.15.78:0/1987215409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638cfed0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 -- 172.21.15.78:0/1987215409 >> 172.21.15.78:0/1987215409 conn(0x55dd638cc060 msgr2=0x55dd638cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 -- 172.21.15.78:0/1987215409 shutdown_connections 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.341+0000 7fe82461d740 1 -- 172.21.15.78:0/1987215409 wait complete. 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 -- start start 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 Processor -- start 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638d8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55dd638dd880 con 0x55dd638cfaf0 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dd637bd290 con 0x55dd638d0410 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dd638dda00 con 0x55dd638dacb0 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe82461d740 10 client.0 _ll_register_callbacks cb 0x55dd637ee140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39926/0 (socket says 172.21.15.78:39926) 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe81a7fc640 1 -- 172.21.15.78:0/3886741877 learned_addr learned my addr 172.21.15.78:0/3886741877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.342+0000 7fe81affd640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638d8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 -- 172.21.15.78:0/3886741877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 msgr2=0x55dd638d8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638d8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 -- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 msgr2=0x55dd638d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 -- 172.21.15.78:0/3886741877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd63547210 con 0x55dd638cfaf0 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81affd640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe810013f50 tx=0x7fe810005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cca7e89517e95f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe7feff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81002e060 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8100047a0 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.343+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe810005460 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe810011780 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 10 client.15852 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd63613620 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe810011e00 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe810033a50 con 0x55dd638cfaf0 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 10 client.15852 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 20 client.15852 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:48.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd638ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 4 client.15852 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.344+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe8100047a0 con 0x55dd638eab30 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.345+0000 7fe81affd640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd638ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.345+0000 7fe81affd640 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd638ecf20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f005d24bdc4f764f server_cookie=9f022c5713540671 in_seq=0 out_seq=0 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.345+0000 7fe7feff5640 10 client.15852 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe7feff5640 1 -- 172.21.15.78:0/3886741877 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fe8100047a0 con 0x55dd638eab30 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe7feff5640 10 client.15852 handle_command_reply: tid=0 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 client.15852 shutdown 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 msgr2=0x55dd638d7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe810013f50 tx=0x7fe810005850 comp rx=0 tx=0).stop 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 shutdown_connections 2024-09-16T01:14:48.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd638ecf20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe7feff5640 0 client.15852 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638d8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.347+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.348+0000 7fe82461d740 1 --2- 172.21.15.78:0/3886741877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d7be0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.348+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 >> 172.21.15.78:0/3886741877 conn(0x55dd638cc060 msgr2=0x55dd638d1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.348+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 shutdown_connections 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.348+0000 7fe82461d740 1 -- 172.21.15.78:0/3886741877 wait complete. 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.348+0000 7fe82461d740 20 client.15852 trim_cache size 0 max 16384 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.369+0000 7fe82461d740 1 Processor -- start 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.369+0000 7fe82461d740 1 -- start start 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638eff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1af590 con 0x55dd638cfaf0 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ace60 con 0x55dd638d0410 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe82461d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1aacb0 con 0x55dd638dacb0 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39932/0 (socket says 172.21.15.78:39932) 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 -- 172.21.15.78:0/4182986047 learned_addr learned my addr 172.21.15.78:0/4182986047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81affd640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 -- 172.21.15.78:0/4182986047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 msgr2=0x55dd638eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 -- 172.21.15.78:0/4182986047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 msgr2=0x55dd638d8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81a7fc640 1 -- 172.21.15.78:0/4182986047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd638cf920 con 0x55dd638cfaf0 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.370+0000 7fe81affd640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.371+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe810015040 tx=0x7fe810033f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b289667beb42694 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.371+0000 7fe800ff9640 1 -- 172.21.15.78:0/4182986047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810039030 con 0x55dd638cfaf0 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.371+0000 7fe800ff9640 1 -- 172.21.15.78:0/4182986047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe810011e00 con 0x55dd638cfaf0 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.371+0000 7fe800ff9640 1 -- 172.21.15.78:0/4182986047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810005460 con 0x55dd638cfaf0 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.372+0000 7fe82461d740 1 -- 172.21.15.78:0/4182986047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 msgr2=0x55dd638d2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.372+0000 7fe82461d740 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe810015040 tx=0x7fe810033f90 comp rx=0 tx=0).stop 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 -- 172.21.15.78:0/4182986047 shutdown_connections 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd638eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638d0410 0x55dd638d8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 --2- 172.21.15.78:0/4182986047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638cfaf0 0x55dd638d2560 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 -- 172.21.15.78:0/4182986047 >> 172.21.15.78:0/4182986047 conn(0x55dd638cc060 msgr2=0x55dd638d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 -- 172.21.15.78:0/4182986047 shutdown_connections 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 -- 172.21.15.78:0/4182986047 wait complete. 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.373+0000 7fe82461d740 1 -- start start 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.374+0000 7fe82461d740 1 Processor -- start 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.374+0000 7fe82461d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.374+0000 7fe82461d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 0x55dd638eda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.374+0000 7fe82461d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd63953460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.374+0000 7fe82461d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ab320 con 0x55dd638d0410 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 0x55dd638eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:49530/0 (socket says 172.21.15.78:49530) 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 0x55dd638eda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39934/0 (socket says 172.21.15.78:39934) 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 -- 172.21.15.78:0/959245665 learned_addr learned my addr 172.21.15.78:0/959245665 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ac790 con 0x55dd638cfaf0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1ab9f0 con 0x55dd638dacb0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe82461d740 10 client.0 _ll_register_callbacks cb 0x7fe81000be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd63953460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 -- 172.21.15.78:0/959245665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 msgr2=0x55dd63953460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd63953460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 -- 172.21.15.78:0/959245665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 msgr2=0x55dd638eda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 0x55dd638eda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd638c72d0 con 0x55dd638cfaf0 2024-09-16T01:14:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe81a7fc640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe810012010 tx=0x7fe810033ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b11ffb3110cb720a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe7feff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810039070 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe810011e00 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.375+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fe810011920 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe81002e040 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 10 client.25316 fetch_fsmap learned FSMap version 31 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dd63954a20 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe810011e00 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe810033480 con 0x55dd638cfaf0 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 10 client.25316 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:14:48.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 10 client.25316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 20 client.25316 populate_metadata read hostname 'smithi078' 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd63956fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 4 client.25316 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd63841a20 con 0x55dd638eab30 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.376+0000 7fe81affd640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd63956fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.377+0000 7fe7feff5640 10 client.25316 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.377+0000 7fe81affd640 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd63956fe0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2de0bd9345462 server_cookie=bd78a3bb80ab57f in_seq=0 out_seq=0 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe7feff5640 1 -- 172.21.15.78:0/959245665 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd63841a20 con 0x55dd638eab30 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe7feff5640 10 client.25316 handle_command_reply: tid=0 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 client.25316 shutdown 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 msgr2=0x55dd638ed4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe810012010 tx=0x7fe810033ef0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 shutdown_connections 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55dd638eab30 0x55dd63956fe0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dd638dacb0 0x55dd63953460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd638d0410 0x55dd638eda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 --2- 172.21.15.78:0/959245665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd638cfaf0 0x55dd638ed4e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 >> 172.21.15.78:0/959245665 conn(0x55dd638cc060 msgr2=0x55dd638e5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe7feff5640 0 client.25316 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 shutdown_connections 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 1 -- 172.21.15.78:0/959245665 wait complete. 2024-09-16T01:14:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.384+0000 7fe82461d740 20 client.25316 trim_cache size 0 max 16384 2024-09-16T01:14:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:14:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:14:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:14:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 msgr2=0x7fe8040a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 0x7fe8040a5a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe8140cd490 tx=0x7fe8140c6d40 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 msgr2=0x7fe81c19cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 0x7fe81c19cc40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400ee30 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 shutdown_connections 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe8040a35c0 0x7fe8040a5a80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe81c107ad0 0x7fe81c19cc40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c102460 0x7fe81c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 --2- 172.21.15.78:0/3463165867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe81c101ac0 0x7fe81c198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 >> 172.21.15.78:0/3463165867 conn(0x7fe81c0fd820 msgr2=0x7fe81c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 shutdown_connections 2024-09-16T01:14:48.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:14:48.385+0000 7fe8217a8640 1 -- 172.21.15.78:0/3463165867 wait complete. 2024-09-16T01:14:49.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:49 smithi148 ceph-mon[35030]: pgmap v123: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 519 op/s 2024-09-16T01:14:49.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:49 smithi078 ceph-mon[30937]: pgmap v123: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 519 op/s 2024-09-16T01:14:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:49 smithi193 ceph-mon[35701]: pgmap v123: 129 pgs: 129 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 519 op/s 2024-09-16T01:14:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 215.0 s, 1794.8 tps, lat 56.311 ms stddev 64.503 2024-09-16T01:14:51.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:51 smithi078 ceph-mon[30937]: pgmap v124: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 795 op/s 2024-09-16T01:14:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:51 smithi193 ceph-mon[35701]: pgmap v124: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 795 op/s 2024-09-16T01:14:51.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:51 smithi148 ceph-mon[35030]: pgmap v124: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 795 op/s 2024-09-16T01:14:53.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:53 smithi078 ceph-mon[30937]: pgmap v125: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 525 op/s 2024-09-16T01:14:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:53 smithi193 ceph-mon[35701]: pgmap v125: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 525 op/s 2024-09-16T01:14:53.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:53 smithi148 ceph-mon[35030]: pgmap v125: 129 pgs: 129 active+clean; 2.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 525 op/s 2024-09-16T01:14:54.315 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.8 2024-09-16T01:14:54.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:14:54.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:54 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:14:54.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:54 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:14:54.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:54 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:14:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:14:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:55 smithi078 ceph-mon[30937]: pgmap v126: 129 pgs: 129 active+clean; 2.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 576 op/s 2024-09-16T01:14:55.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:14:55.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:14:55.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:14:55.536 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:55 smithi148 ceph-mon[35030]: pgmap v126: 129 pgs: 129 active+clean; 2.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 576 op/s 2024-09-16T01:14:55.536 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:14:55.536 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:14:55.536 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:14:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:55 smithi193 ceph-mon[35701]: pgmap v126: 129 pgs: 129 active+clean; 2.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 MiB/s wr, 576 op/s 2024-09-16T01:14:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:14:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:14:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:14:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 220.0 s, 1697.6 tps, lat 58.861 ms stddev 68.611 2024-09-16T01:14:57.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:57 smithi078 ceph-mon[30937]: pgmap v127: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-09-16T01:14:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:57 smithi193 ceph-mon[35701]: pgmap v127: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-09-16T01:14:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:57 smithi148 ceph-mon[35030]: pgmap v127: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-09-16T01:14:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 -- 172.21.15.78:0/4010321436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c06e4f0 msgr2=0x7ffb1c06e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 --2- 172.21.15.78:0/4010321436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c06e8d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009cd0 tx=0x7ffb1402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 -- 172.21.15.78:0/4010321436 shutdown_connections 2024-09-16T01:14:57.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 --2- 172.21.15.78:0/4010321436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c114ab0 0x7ffb1c116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 --2- 172.21.15.78:0/4010321436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 --2- 172.21.15.78:0/4010321436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c06e8d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.858+0000 7ffb21f05640 1 -- 172.21.15.78:0/4010321436 >> 172.21.15.78:0/4010321436 conn(0x7ffb1c073180 msgr2=0x7ffb1c073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.860+0000 7ffb21f05640 1 -- 172.21.15.78:0/4010321436 shutdown_connections 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.860+0000 7ffb21f05640 1 -- 172.21.15.78:0/4010321436 wait complete. 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 Processor -- start 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 -- start start 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c114ab0 0x7ffb1c1a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c1183e0 con 0x7ffb1c06e4f0 2024-09-16T01:14:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c118260 con 0x7ffb1c114ab0 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21f05640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c118560 con 0x7ffb1c06ee10 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb20f03640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb20f03640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39872/0 (socket says 172.21.15.78:39872) 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb1bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:38994/0 (socket says 172.21.15.78:38994) 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb20f03640 1 -- 172.21.15.78:0/3098324515 learned_addr learned my addr 172.21.15.78:0/3098324515 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.861+0000 7ffb21704640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c114ab0 0x7ffb1c1a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 -- 172.21.15.78:0/3098324515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c114ab0 msgr2=0x7ffb1c1a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c114ab0 0x7ffb1c1a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 -- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 msgr2=0x7ffb1c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 -- 172.21.15.78:0/3098324515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1c1a38d0 con 0x7ffb1c06ee10 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb1bfff640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000e9f0 tx=0x7ffb1000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1000ccc0 con 0x7ffb1c06ee10 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb10005020 con 0x7ffb1c06ee10 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.862+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10004b20 con 0x7ffb1c06ee10 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.863+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1c1a3b60 con 0x7ffb1c06ee10 2024-09-16T01:14:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.863+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb1c1ab520 con 0x7ffb1c06ee10 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.865+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb1c1096f0 con 0x7ffb1c06ee10 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.865+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7ffb10004cc0 con 0x7ffb1c06ee10 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.866+0000 7ffb19ffb640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 0x7ffb040a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.867+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffb100cf650 con 0x7ffb1c06ee10 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.867+0000 7ffb20f03640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 0x7ffb040a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:14:57.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.867+0000 7ffb20f03640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 0x7ffb040a5a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb14000f80 tx=0x7ffb14009800 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:14:57.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:57.870+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffb1009bc40 con 0x7ffb1c06ee10 2024-09-16T01:14:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.276+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb1c19fc80 con 0x7ffb1c06ee10 2024-09-16T01:14:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.281+0000 7ffb19ffb640 1 -- 172.21.15.78:0/3098324515 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7ffb1009b540 con 0x7ffb1c06ee10 2024-09-16T01:14:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:14:58.284 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:14:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:14:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.286+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 msgr2=0x7ffb040a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.286+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 0x7ffb040a5a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb14000f80 tx=0x7ffb14009800 comp rx=0 tx=0).stop 2024-09-16T01:14:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.286+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 msgr2=0x7ffb1c19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:14:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.286+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000e9f0 tx=0x7ffb1000eeb0 comp rx=0 tx=0).stop 2024-09-16T01:14:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 shutdown_connections 2024-09-16T01:14:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb040a3560 0x7ffb040a5a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c114ab0 0x7ffb1c1a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb1c06ee10 0x7ffb1c19eeb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:58.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 --2- 172.21.15.78:0/3098324515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb1c06e4f0 0x7ffb1c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:14:58.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.288+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 >> 172.21.15.78:0/3098324515 conn(0x7ffb1c073180 msgr2=0x7ffb1c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:14:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.293+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 shutdown_connections 2024-09-16T01:14:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:14:58.294+0000 7ffb21f05640 1 -- 172.21.15.78:0/3098324515 wait complete. 2024-09-16T01:14:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:59 smithi078 ceph-mon[30937]: pgmap v128: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-09-16T01:14:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:14:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3098324515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:14:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:59 smithi193 ceph-mon[35701]: pgmap v128: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-09-16T01:14:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:14:59 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3098324515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:14:59.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:59 smithi148 ceph-mon[35030]: pgmap v128: 129 pgs: 129 active+clean; 2.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-09-16T01:14:59.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:14:59 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3098324515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:14:59.655 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0' 2024-09-16T01:14:59.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0 2024-09-16T01:15:00.059 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:15:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 225.0 s, 1488.0 tps, lat 67.092 ms stddev 109.223 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 -- 172.21.15.78:0/1148721381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2398101350 msgr2=0x7f239810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 --2- 172.21.15.78:0/1148721381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2398101350 0x7f239810d880 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2390009cd0 tx=0x7f239002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 -- 172.21.15.78:0/1148721381 shutdown_connections 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 --2- 172.21.15.78:0/1148721381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2398101350 0x7f239810d880 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 --2- 172.21.15.78:0/1148721381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23981009b0 0x7f2398100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 --2- 172.21.15.78:0/1148721381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981069c0 0x7f2398106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 -- 172.21.15.78:0/1148721381 >> 172.21.15.78:0/1148721381 conn(0x7f2398077990 msgr2=0x7f23980fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.498+0000 7f239eaba640 1 -- 172.21.15.78:0/1148721381 shutdown_connections 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 -- 172.21.15.78:0/1148721381 wait complete. 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 Processor -- start 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 -- start start 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 0x7f2398199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2398101350 0x7f23981996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2398110e00 con 0x7f23981069c0 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2398110c80 con 0x7f2398101350 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239eaba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2398110f80 con 0x7f23981009b0 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239d057640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 0x7f2398199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239d057640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39894/0 (socket says 172.21.15.78:39894) 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.499+0000 7f239d057640 1 -- 172.21.15.78:0/3632721736 learned_addr learned my addr 172.21.15.78:0/3632721736 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f2397fff640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2398101350 0x7f23981996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 -- 172.21.15.78:0/3632721736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 msgr2=0x7f2398199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 0x7f2398199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 -- 172.21.15.78:0/3632721736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2398101350 msgr2=0x7f23981996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2398101350 0x7f23981996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 -- 172.21.15.78:0/3632721736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23981a4730 con 0x7f23981069c0 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239c856640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 0x7f2398199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239d057640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2390005cc0 tx=0x7f2390032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f2395ffb640 1 -- 172.21.15.78:0/3632721736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2390004040 con 0x7f23981069c0 2024-09-16T01:15:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23981a4900 con 0x7f23981069c0 2024-09-16T01:15:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f2395ffb640 1 -- 172.21.15.78:0/3632721736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23900046c0 con 0x7f23981069c0 2024-09-16T01:15:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f2395ffb640 1 -- 172.21.15.78:0/3632721736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2390030e00 con 0x7f23981069c0 2024-09-16T01:15:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.500+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2398196f60 con 0x7f23981069c0 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.503+0000 7f2395ffb640 1 -- 172.21.15.78:0/3632721736 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f23900305a0 con 0x7f23981069c0 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f2395ffb640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 0x7f23880a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 Processor -- start 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 -- start start 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f239c856640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 0x7f23880a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f2395ffb640 1 -- 172.21.15.78:0/3632721736 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f239002e030 con 0x7f23981069c0 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 0x5613fe064770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe064cb0 0x5613fe0670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5613fdfbe2f0 con 0x5613fe05a410 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5613fdf66270 con 0x5613fe064cb0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f23a185a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613fdfba2b0 con 0x5613fe059af0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:39020/0 (socket says 172.21.15.78:39020) 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f239c856640 1 -- 172.21.15.78:0/1246264783 learned_addr learned my addr 172.21.15.78:0/1246264783 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f2397fff640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 0x5613fe064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.504+0000 7f239d057640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe064cb0 0x5613fe0670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 0x7f23880a9d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f238c011930 tx=0x7f238c011d30 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 -- 172.21.15.78:0/1246264783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe064cb0 msgr2=0x5613fe0670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe064cb0 0x5613fe0670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 -- 172.21.15.78:0/1246264783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 msgr2=0x5613fe064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 0x5613fe064770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 -- 172.21.15.78:0/1246264783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613fdf65340 con 0x5613fe059af0 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f2397fff640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 0x5613fe064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f239c856640 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f238c00fc80 tx=0x7f238c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f0e89666a84933b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f2382ffd640 1 -- 172.21.15.78:0/1246264783 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c01a040 con 0x5613fe059af0 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f2382ffd640 1 -- 172.21.15.78:0/1246264783 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238c024070 con 0x5613fe059af0 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.505+0000 7f2382ffd640 1 -- 172.21.15.78:0/1246264783 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c003c40 con 0x5613fe059af0 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- 172.21.15.78:0/1246264783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 msgr2=0x5613fe059ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f238c00fc80 tx=0x7f238c020000 comp rx=0 tx=0).stop 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- 172.21.15.78:0/1246264783 shutdown_connections 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe064cb0 0x5613fe0670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe05a410 0x5613fe064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 --2- 172.21.15.78:0/1246264783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe059ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- 172.21.15.78:0/1246264783 >> 172.21.15.78:0/1246264783 conn(0x5613fe056060 msgr2=0x5613fe056440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- 172.21.15.78:0/1246264783 shutdown_connections 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- 172.21.15.78:0/1246264783 wait complete. 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 -- start start 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.506+0000 7f23a185a740 1 Processor -- start 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe061a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe061fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe062510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5613fe067880 con 0x5613fe064cb0 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5613fdf47290 con 0x5613fe05a410 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613fe067a00 con 0x5613fe059af0 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f23a185a740 10 client.0 _ll_register_callbacks cb 0x5613fdf78140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239d057640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe062510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe061a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239d057640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe062510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39922/0 (socket says 172.21.15.78:39922) 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239d057640 1 -- 172.21.15.78:0/2884038222 learned_addr learned my addr 172.21.15.78:0/2884038222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f2397fff640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe061fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 -- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 msgr2=0x5613fe061fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe061fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 -- 172.21.15.78:0/2884038222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 msgr2=0x5613fe062510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe062510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 -- 172.21.15.78:0/2884038222 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613fdcd1210 con 0x5613fe059af0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f239c856640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe061a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f238c00f790 tx=0x7f238c005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9435de8468a8e0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.507+0000 7f2377fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c038060 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f238c005250 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f238c024070 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f238c01a070 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f23a185a740 10 client.25384 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5613fdd9d620 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.508+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238c024070 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f238c01d030 con 0x5613fe059af0 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 10 client.25384 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 10 client.25384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 20 client.25384 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe074a40 0x5613fe076e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 4 client.25384 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613fdfcba20 con 0x5613fe074a40 2024-09-16T01:15:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f2397fff640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe074a40 0x5613fe076e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f2377fff640 10 client.25384 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.509+0000 7f2397fff640 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe074a40 0x5613fe076e30 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da50d7d479ee7e7 server_cookie=346b22acf9b74613 in_seq=0 out_seq=0 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f2377fff640 1 -- 172.21.15.78:0/2884038222 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5613fdfcba20 con 0x5613fe074a40 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f2377fff640 10 client.25384 handle_command_reply: tid=0 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 client.25384 shutdown 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 msgr2=0x5613fe061a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe061a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f238c00f790 tx=0x7f238c005d30 comp rx=0 tx=0).stop 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 shutdown_connections 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe074a40 0x5613fe076e30 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe062510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f2377fff640 0 client.25384 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe061fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 --2- 172.21.15.78:0/2884038222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe061a90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.510+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 >> 172.21.15.78:0/2884038222 conn(0x5613fe056060 msgr2=0x5613fe05bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.511+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 shutdown_connections 2024-09-16T01:15:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.511+0000 7f23a185a740 1 -- 172.21.15.78:0/2884038222 wait complete. 2024-09-16T01:15:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.511+0000 7f23a185a740 20 client.25384 trim_cache size 0 max 16384 2024-09-16T01:15:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:01 smithi078 ceph-mon[30937]: pgmap v129: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-09-16T01:15:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 Processor -- start 2024-09-16T01:15:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 -- start start 2024-09-16T01:15:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe064cb0 0x5613fe07c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f23981af560 con 0x5613fe059af0 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f23981ace30 con 0x5613fe05a410 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f23a185a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23981aac80 con 0x5613fe064cb0 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39934/0 (socket says 172.21.15.78:39934) 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 -- 172.21.15.78:0/1386076625 learned_addr learned my addr 172.21.15.78:0/1386076625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239d057640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe064cb0 0x5613fe07c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f2397fff640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 -- 172.21.15.78:0/1386076625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe064cb0 msgr2=0x5613fe07c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe064cb0 0x5613fe07c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 -- 172.21.15.78:0/1386076625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 msgr2=0x5613fe0550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f239c856640 1 -- 172.21.15.78:0/1386076625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613fe05a240 con 0x5613fe059af0 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.528+0000 7f2397fff640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.529+0000 7f239c856640 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f238c004f10 tx=0x7f238c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c0ff9960962b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.529+0000 7f23827fc640 1 -- 172.21.15.78:0/1386076625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c038040 con 0x5613fe059af0 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.529+0000 7f23827fc640 1 -- 172.21.15.78:0/1386076625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238c024040 con 0x5613fe059af0 2024-09-16T01:15:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.529+0000 7f23827fc640 1 -- 172.21.15.78:0/1386076625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c020e60 con 0x5613fe059af0 2024-09-16T01:15:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 -- 172.21.15.78:0/1386076625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 msgr2=0x5613fe062340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f238c004f10 tx=0x7f238c005f10 comp rx=0 tx=0).stop 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 -- 172.21.15.78:0/1386076625 shutdown_connections 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe064cb0 0x5613fe07c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 --2- 172.21.15.78:0/1386076625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe059af0 0x5613fe062340 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 -- 172.21.15.78:0/1386076625 >> 172.21.15.78:0/1386076625 conn(0x5613fe056060 msgr2=0x5613fe0549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.530+0000 7f23a185a740 1 -- 172.21.15.78:0/1386076625 shutdown_connections 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.531+0000 7f23a185a740 1 -- 172.21.15.78:0/1386076625 wait complete. 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.531+0000 7f23a185a740 1 -- start start 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.531+0000 7f23a185a740 1 Processor -- start 2024-09-16T01:15:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe066a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f23981ab2f0 con 0x5613fe064cb0 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f23981ac760 con 0x5613fe05a410 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23981ab9c0 con 0x5613fe059af0 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f23a185a740 10 client.0 _ll_register_callbacks cb 0x7f238c00fbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239d057640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe066a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:39054/0 (socket says 172.21.15.78:39054) 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 -- 172.21.15.78:0/1176666658 learned_addr learned my addr 172.21.15.78:0/1176666658 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:01.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f2397fff640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 -- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 msgr2=0x5613fe0664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0664c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 -- 172.21.15.78:0/1176666658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 msgr2=0x5613fe066a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe066a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 -- 172.21.15.78:0/1176666658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613fe06b5a0 con 0x5613fe059af0 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f2397fff640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239d057640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe066a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f239c856640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f238c02d040 tx=0x7f238c003a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=984575f10f70d266 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.532+0000 7f2377fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c048060 con 0x5613fe059af0 2024-09-16T01:15:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f238c040040 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f238c024040 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f238c04f050 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 10 client.25390 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5613fe0df800 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238c024040 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f238c034740 con 0x5613fe059af0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 10 client.25390 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 10 client.25390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 20 client.25390 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe0dfeb0 0x5613fe0e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 4 client.25390 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613fdfc5d70 con 0x5613fe0dfeb0 2024-09-16T01:15:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.533+0000 7f2397fff640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe0dfeb0 0x5613fe0e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.534+0000 7f2377fff640 10 client.25390 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.534+0000 7f2397fff640 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe0dfeb0 0x5613fe0e22d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f4616d2881a4d9 server_cookie=1c1adf74601d93a8 in_seq=0 out_seq=0 2024-09-16T01:15:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:01 smithi193 ceph-mon[35701]: pgmap v129: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-09-16T01:15:01.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:01 smithi148 ceph-mon[35030]: pgmap v129: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-09-16T01:15:01.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.698+0000 7f2377fff640 1 -- 172.21.15.78:0/1176666658 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613fdfc5d70 con 0x5613fe0dfeb0 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.698+0000 7f2377fff640 10 client.25390 handle_command_reply: tid=0 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.698+0000 7f23a185a740 1 client.25390 shutdown 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 msgr2=0x5613fe065f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f238c02d040 tx=0x7f238c003a30 comp rx=0 tx=0).stop 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 shutdown_connections 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5613fe0dfeb0 0x5613fe0e22d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5613fe064cb0 0x5613fe066a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f2377fff640 0 client.25390 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5613fe05a410 0x5613fe0664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 --2- 172.21.15.78:0/1176666658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613fe059af0 0x5613fe065f80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 >> 172.21.15.78:0/1176666658 conn(0x5613fe056060 msgr2=0x5613fe0551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 shutdown_connections 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 1 -- 172.21.15.78:0/1176666658 wait complete. 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f23a185a740 20 client.25390 trim_cache size 0 max 16384 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:15:01.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:15:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 msgr2=0x7f23880a9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 0x7f23880a9d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f238c011930 tx=0x7f238c011d30 comp rx=0 tx=0).stop 2024-09-16T01:15:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 msgr2=0x7f23981a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.699+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2390005cc0 tx=0x7f2390032f40 comp rx=0 tx=0).stop 2024-09-16T01:15:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 shutdown_connections 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f23880a7870 0x7f23880a9d30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23981069c0 0x7f23981a41f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2398101350 0x7f23981996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 --2- 172.21.15.78:0/3632721736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23981009b0 0x7f2398199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 >> 172.21.15.78:0/3632721736 conn(0x7f2398077990 msgr2=0x7f2398103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 shutdown_connections 2024-09-16T01:15:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:01.700+0000 7f239eaba640 1 -- 172.21.15.78:0/3632721736 wait complete. 2024-09-16T01:15:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:03 smithi078 ceph-mon[30937]: pgmap v130: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-09-16T01:15:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:03 smithi193 ceph-mon[35701]: pgmap v130: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-09-16T01:15:03.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:03 smithi148 ceph-mon[35030]: pgmap v130: 129 pgs: 129 active+clean; 2.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-09-16T01:15:04.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 -- 172.21.15.78:0/2459711686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a81002f0 msgr2=0x7f01a81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 --2- 172.21.15.78:0/2459711686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a81002f0 0x7f01a81006d0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0198009930 tx=0x7f019802a420 comp rx=0 tx=0).stop 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 -- 172.21.15.78:0/2459711686 shutdown_connections 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 --2- 172.21.15.78:0/2459711686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a8106f50 0x7f01a810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 --2- 172.21.15.78:0/2459711686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8100ca0 0x7f01a8106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 --2- 172.21.15.78:0/2459711686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a81002f0 0x7f01a81006d0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 -- 172.21.15.78:0/2459711686 >> 172.21.15.78:0/2459711686 conn(0x7f01a8077220 msgr2=0x7f01a80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.354+0000 7f01adba0640 1 -- 172.21.15.78:0/2459711686 shutdown_connections 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 -- 172.21.15.78:0/2459711686 wait complete. 2024-09-16T01:15:04.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 Processor -- start 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 -- start start 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 0x7f01a8101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8106f50 0x7f01a8101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.355+0000 7f01adba0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01a8111ff0 con 0x7f01a8100ca0 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01adba0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f01a8111e70 con 0x7f01a8106f50 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01adba0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01a8112170 con 0x7f01a81002f0 2024-09-16T01:15:04.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 0x7f01a8101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37530/0 (socket says 172.21.15.78:37530) 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3212007961 learned_addr learned my addr 172.21.15.78:0/3212007961 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 0x7f01a8101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:44490/0 (socket says 172.21.15.78:44490) 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a7fff640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8106f50 0x7f01a8101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3212007961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 msgr2=0x7f01a8101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 0x7f01a8101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3212007961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8106f50 msgr2=0x7f01a8101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8106f50 0x7f01a8101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3212007961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a81a69b0 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.356+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e9c0 tx=0x7f019c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.357+0000 7f01a4ff9640 1 -- 172.21.15.78:0/3212007961 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c00cd80 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.357+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a81a6ca0 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.357+0000 7f01a4ff9640 1 -- 172.21.15.78:0/3212007961 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f019c005500 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.357+0000 7f01a4ff9640 1 -- 172.21.15.78:0/3212007961 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c010690 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.357+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a81a71d0 con 0x7f01a8100ca0 2024-09-16T01:15:04.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.359+0000 7f01aff79740 1 Processor -- start 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.359+0000 7f01aff79740 1 -- start start 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d25b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d30400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d32d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c917c8b2a0 con 0x55c917d26080 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c917c32b10 con 0x55c917d30940 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01aff79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c917c87260 con 0x55c917d25760 2024-09-16T01:15:04.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d30400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d30400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37536/0 (socket says 172.21.15.78:37536) 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a77fe640 1 -- 172.21.15.78:0/2860561959 learned_addr learned my addr 172.21.15.78:0/2860561959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d25b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 -- 172.21.15.78:0/2860561959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 msgr2=0x55c917d32d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d32d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 -- 172.21.15.78:0/2860561959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 msgr2=0x55c917d30400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d30400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 -- 172.21.15.78:0/2860561959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9179fcb90 con 0x55c917d25760 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.360+0000 7f01a7fff640 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d25b40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009f00 tx=0x7f01a0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a05e33b50fb34302 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.361+0000 7f018dffb640 1 -- 172.21.15.78:0/2860561959 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0004d20 con 0x55c917d25760 2024-09-16T01:15:04.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.361+0000 7f018dffb640 1 -- 172.21.15.78:0/2860561959 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a0004050 con 0x55c917d25760 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.361+0000 7f01a4ff9640 1 -- 172.21.15.78:0/3212007961 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f019c0046c0 con 0x7f01a8100ca0 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.361+0000 7f018dffb640 1 -- 172.21.15.78:0/2860561959 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0005260 con 0x55c917d25760 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- 172.21.15.78:0/2860561959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 msgr2=0x55c917d25b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d25b40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009f00 tx=0x7f01a0011810 comp rx=0 tx=0).stop 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- 172.21.15.78:0/2860561959 shutdown_connections 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d32d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d30400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 --2- 172.21.15.78:0/2860561959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d25b40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- 172.21.15.78:0/2860561959 >> 172.21.15.78:0/2860561959 conn(0x55c917d21cf0 msgr2=0x55c917d220d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- 172.21.15.78:0/2860561959 shutdown_connections 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- 172.21.15.78:0/2860561959 wait complete. 2024-09-16T01:15:04.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01a4ff9640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 0x7f01900a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 -- start start 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01a4ff9640 1 -- 172.21.15.78:0/3212007961 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f019c0d0490 con 0x7f01a8100ca0 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 0x7f01900a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.362+0000 7f01aff79740 1 Processor -- start 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d2dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d30940 0x55c917d2e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c917d335b0 con 0x55c917d30940 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c917c136e0 con 0x55c917d26080 2024-09-16T01:15:04.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c917d33730 con 0x55c917d25760 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01aff79740 10 client.0 _ll_register_callbacks cb 0x55c917c37a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d30940 0x55c917d2e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:15:04.965 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 14, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "1b5f538a": { 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 14, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:15:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 133.8, 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:04.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:44500/0 (socket says 172.21.15.78:44500) 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 -- 172.21.15.78:0/173162300 learned_addr learned my addr 172.21.15.78:0/173162300 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d2dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 -- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 msgr2=0x55c917d2dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d2dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 -- 172.21.15.78:0/173162300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d30940 msgr2=0x55c917d2e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d30940 0x55c917d2e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 -- 172.21.15.78:0/173162300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c917d3caa0 con 0x55c917d25760 2024-09-16T01:15:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 0x7f01900a5810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f019802a940 tx=0x7f0198000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f01a77fe640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0198009930 tx=0x7f0198009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d88a3d2155413eca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.363+0000 7f0186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198035040 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f01980048c0 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01a0004050 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f019805b050 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 10 client.25402 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c917d3f430 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0198004cd0 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0198032920 con 0x55c917d25760 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 10 client.25402 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:04.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 10 client.25402 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 20 client.25402 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917d40d70 0x55c917d43210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 4 client.25402 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.364+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f01a0004050 con 0x55c917d40d70 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.365+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917d40d70 0x55c917d43210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.365+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917d40d70 0x55c917d43210 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58354434c17ff74 server_cookie=d2c9821d8bd81dbb in_seq=0 out_seq=0 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.365+0000 7f0186ffd640 10 client.25402 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f0186ffd640 1 -- 172.21.15.78:0/173162300 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f01a0004050 con 0x55c917d40d70 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f0186ffd640 10 client.25402 handle_command_reply: tid=0 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 client.25402 shutdown 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 msgr2=0x55c917d2d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0198009930 tx=0x7f0198009260 comp rx=0 tx=0).stop 2024-09-16T01:15:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 shutdown_connections 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917d40d70 0x55c917d43210 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d30940 0x55c917d2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d2dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f0186ffd640 0 client.25402 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 --2- 172.21.15.78:0/173162300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d2d7b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 >> 172.21.15.78:0/173162300 conn(0x55c917d21cf0 msgr2=0x55c917d20e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 shutdown_connections 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 1 -- 172.21.15.78:0/173162300 wait complete. 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.366+0000 7f01aff79740 20 client.25402 trim_cache size 0 max 16384 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 Processor -- start 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 -- start start 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d20d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d30940 0x55c917d48b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81b3b90 con 0x55c917d25760 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81b1460 con 0x55c917d26080 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.391+0000 7f01aff79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81af2b0 con 0x55c917d30940 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37562/0 (socket says 172.21.15.78:37562) 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 -- 172.21.15.78:0/2019971984 learned_addr learned my addr 172.21.15.78:0/2019971984 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d20d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a7fff640 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d30940 0x55c917d48b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 -- 172.21.15.78:0/2019971984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d30940 msgr2=0x55c917d48b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d30940 0x55c917d48b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 -- 172.21.15.78:0/2019971984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 msgr2=0x55c917d20d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d20d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 -- 172.21.15.78:0/2019971984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c917d21b20 con 0x55c917d25760 2024-09-16T01:15:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f01a77fe640 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0198009230 tx=0x7f0198005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8636505a97116a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f018d7fa640 1 -- 172.21.15.78:0/2019971984 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019803d510 con 0x55c917d25760 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f018d7fa640 1 -- 172.21.15.78:0/2019971984 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0198032aa0 con 0x55c917d25760 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.392+0000 7f018d7fa640 1 -- 172.21.15.78:0/2019971984 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019803d9b0 con 0x55c917d25760 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 -- 172.21.15.78:0/2019971984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 msgr2=0x55c917d281c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0198009230 tx=0x7f0198005410 comp rx=0 tx=0).stop 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 -- 172.21.15.78:0/2019971984 shutdown_connections 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d30940 0x55c917d48b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d26080 0x55c917d20d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 --2- 172.21.15.78:0/2019971984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d25760 0x55c917d281c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 -- 172.21.15.78:0/2019971984 >> 172.21.15.78:0/2019971984 conn(0x55c917d21cf0 msgr2=0x55c917d20650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 -- 172.21.15.78:0/2019971984 shutdown_connections 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.393+0000 7f01aff79740 1 -- 172.21.15.78:0/2019971984 wait complete. 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 -- start start 2024-09-16T01:15:04.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 Processor -- start 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d38be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81af920 con 0x55c917d26080 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81b0d90 con 0x55c917d30940 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81afff0 con 0x55c917d25760 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01aff79740 10 client.0 _ll_register_callbacks cb 0x7f01980041f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37570/0 (socket says 172.21.15.78:37570) 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3282309663 learned_addr learned my addr 172.21.15.78:0/3282309663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.394+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d38be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a7fff640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3282309663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 msgr2=0x55c917d38be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d38be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 msgr2=0x55c917d359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 -- 172.21.15.78:0/3282309663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c917d361b0 con 0x55c917d26080 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d38be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01a6ffd640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f019c0987e0 tx=0x7f019c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f3ef0883d6d336 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f0186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c0a0730 con 0x55c917d26080 2024-09-16T01:15:04.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f019c0989f0 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0198032aa0 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f019c0a4ff0 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01aff79740 10 client.15900 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c917dac5a0 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.395+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0198032aa0 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f019c0a4b60 con 0x55c917d26080 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 10 client.15900 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 10 client.15900 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 20 client.15900 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917dacd60 0x55c917daf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:04.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 4 client.15900 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}] 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}) -- 0x55c917c989d0 con 0x55c917dacd60 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917dacd60 0x55c917daf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f01a77fe640 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917dacd60 0x55c917daf180 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485ac74ade3a0bd9 server_cookie=a62d9790ddbf78ea in_seq=0 out_seq=0 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.396+0000 7f0186ffd640 10 client.15900 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f0186ffd640 1 -- 172.21.15.78:0/3282309663 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c917c989d0 con 0x55c917dacd60 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f0186ffd640 10 client.15900 handle_command_reply: tid=0 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f01aff79740 1 client.15900 shutdown 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 msgr2=0x55c917d39120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f019c0987e0 tx=0x7f019c009310 comp rx=0 tx=0).stop 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.681+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 shutdown_connections 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c917dacd60 0x55c917daf180 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c917d30940 0x55c917d359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c917d26080 0x55c917d39120 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f0186ffd640 0 client.15900 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 --2- 172.21.15.78:0/3282309663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c917d25760 0x55c917d38be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 >> 172.21.15.78:0/3282309663 conn(0x55c917d21cf0 msgr2=0x55c917d20e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 shutdown_connections 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 1 -- 172.21.15.78:0/3282309663 wait complete. 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01aff79740 20 client.15900 trim_cache size 0 max 16384 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 msgr2=0x7f01900a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 0x7f01900a5810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f019802a940 tx=0x7f0198000bc0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 msgr2=0x7f01a81017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e9c0 tx=0x7f019c00ee80 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 shutdown_connections 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f01900a3350 0x7f01900a5810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a8106f50 0x7f01a8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01a8100ca0 0x7f01a81017b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 --2- 172.21.15.78:0/3212007961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01a81002f0 0x7f01a8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:04.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 >> 172.21.15.78:0/3212007961 conn(0x7f01a8077220 msgr2=0x7f01a8105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:04.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.682+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 shutdown_connections 2024-09-16T01:15:04.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:04.683+0000 7f01adba0640 1 -- 172.21.15.78:0/3212007961 wait complete. 2024-09-16T01:15:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:05 smithi078 ceph-mon[30937]: pgmap v131: 129 pgs: 129 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 852 op/s 2024-09-16T01:15:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:05 smithi193 ceph-mon[35701]: pgmap v131: 129 pgs: 129 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 852 op/s 2024-09-16T01:15:05.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:05 smithi148 ceph-mon[35030]: pgmap v131: 129 pgs: 129 active+clean; 2.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 852 op/s 2024-09-16T01:15:05.847 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1b5f538a', quiesce took 11.5 seconds. Will release after: 2.3000000000000007 2024-09-16T01:15:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 230.0 s, 96.8 tps, lat 182.688 ms stddev 170.170 2024-09-16T01:15:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:07 smithi078 ceph-mon[30937]: pgmap v132: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 1.68k op/s 2024-09-16T01:15:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:07 smithi193 ceph-mon[35701]: pgmap v132: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 1.68k op/s 2024-09-16T01:15:07.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:07 smithi148 ceph-mon[35030]: pgmap v132: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 MiB/s wr, 1.68k op/s 2024-09-16T01:15:08.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:15:08.409 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/1390837108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c28106f40 msgr2=0x7f7c2810eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/1390837108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c28106f40 0x7f7c2810eac0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009ca0 tx=0x7f7c2002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/1390837108 shutdown_connections 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/1390837108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c28106f40 0x7f7c2810eac0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/1390837108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c281027c0 0x7f7c28106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/1390837108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 0x7f7c281021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/1390837108 >> 172.21.15.78:0/1390837108 conn(0x7f7c280fd8d0 msgr2=0x7f7c280ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/1390837108 shutdown_connections 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.889+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/1390837108 wait complete. 2024-09-16T01:15:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 Processor -- start 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 -- start start 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 0x7f7c28198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c28106f40 0x7f7c2819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28111fc0 con 0x7f7c281027c0 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28111e40 con 0x7f7c28101e10 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.890+0000 7f7c2f3e7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28112140 con 0x7f7c28106f40 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37596/0 (socket says 172.21.15.78:37596) 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 -- 172.21.15.78:0/3435839796 learned_addr learned my addr 172.21.15.78:0/3435839796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2d183640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 0x7f7c28198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2d984640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c28106f40 0x7f7c2819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 -- 172.21.15.78:0/3435839796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c28106f40 msgr2=0x7f7c2819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c28106f40 0x7f7c2819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 -- 172.21.15.78:0/3435839796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 msgr2=0x7f7c28198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 0x7f7c28198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 -- 172.21.15.78:0/3435839796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2819d3f0 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2c982640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800b4a0 tx=0x7f7c1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18014070 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2819d6e0 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c18010af0 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18017d10 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.891+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c281a4fc0 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.892+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c281021f0 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.893+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7c18017070 con 0x7f7c281027c0 2024-09-16T01:15:08.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.894+0000 7f7c1e7fc640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 0x7f7c140a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:08.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.894+0000 7f7c2d183640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 0x7f7c140a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:08.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.894+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f7c180cf370 con 0x7f7c281027c0 2024-09-16T01:15:08.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.894+0000 7f7c2d183640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 0x7f7c140a5870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400bbe0 tx=0x7f7c2400b390 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:08.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:08.898+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7c1809bad0 con 0x7f7c281027c0 2024-09-16T01:15:09.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.099+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2802d7c0 con 0x7f7c281027c0 2024-09-16T01:15:09.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.099+0000 7f7c1e7fc640 1 -- 172.21.15.78:0/3435839796 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f7c1809b3d0 con 0x7f7c281027c0 2024-09-16T01:15:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:15:09.102 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:15:09.102 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 msgr2=0x7f7c140a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 0x7f7c140a5870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400bbe0 tx=0x7f7c2400b390 comp rx=0 tx=0).stop 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 msgr2=0x7f7c28198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800b4a0 tx=0x7f7c1800b960 comp rx=0 tx=0).stop 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 shutdown_connections 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7c140a33b0 0x7f7c140a5870 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c28106f40 0x7f7c2819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.101+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c281027c0 0x7f7c28198980 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:09.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.102+0000 7f7c2f3e7640 1 --2- 172.21.15.78:0/3435839796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c28101e10 0x7f7c28198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:09.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.102+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 >> 172.21.15.78:0/3435839796 conn(0x7f7c280fd8d0 msgr2=0x7f7c280fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:09.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.102+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 shutdown_connections 2024-09-16T01:15:09.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:09.102+0000 7f7c2f3e7640 1 -- 172.21.15.78:0/3435839796 wait complete. 2024-09-16T01:15:09.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:09 smithi078 ceph-mon[30937]: pgmap v133: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-09-16T01:15:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:09 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3435839796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:09 smithi193 ceph-mon[35701]: pgmap v133: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-09-16T01:15:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:09 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3435839796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:09 smithi148 ceph-mon[35030]: pgmap v133: 129 pgs: 129 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-09-16T01:15:09.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:09 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3435839796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:09.943 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 1b5f538a --release --await' 2024-09-16T01:15:09.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 1b5f538a --release --await 2024-09-16T01:15:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 -- 172.21.15.78:0/1828371506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4102d40 msgr2=0x7f33c4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 --2- 172.21.15.78:0/1828371506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4102d40 0x7f33c4103120 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009900 tx=0x7f33c002a480 comp rx=0 tx=0).stop 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 -- 172.21.15.78:0/1828371506 shutdown_connections 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 --2- 172.21.15.78:0/1828371506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c410c170 0x7f33c410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 --2- 172.21.15.78:0/1828371506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c4103660 0x7f33c410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 --2- 172.21.15.78:0/1828371506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4102d40 0x7f33c4103120 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 -- 172.21.15.78:0/1828371506 >> 172.21.15.78:0/1828371506 conn(0x7f33c40fc7a0 msgr2=0x7f33c40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 -- 172.21.15.78:0/1828371506 shutdown_connections 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.680+0000 7f33cc288640 1 -- 172.21.15.78:0/1828371506 wait complete. 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 Processor -- start 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 -- start start 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 0x7f33c41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c410c170 0x7f33c419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4111f70 con 0x7f33c4103660 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4111df0 con 0x7f33c410c170 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33cc288640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41120f0 con 0x7f33c4102d40 2024-09-16T01:15:10.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37614/0 (socket says 172.21.15.78:37614) 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 -- 172.21.15.78:0/2467915932 learned_addr learned my addr 172.21.15.78:0/2467915932 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33ca024640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 0x7f33c41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33ca825640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c410c170 0x7f33c419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 -- 172.21.15.78:0/2467915932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 msgr2=0x7f33c41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 0x7f33c41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 -- 172.21.15.78:0/2467915932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c410c170 msgr2=0x7f33c419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c410c170 0x7f33c419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33c9823640 1 -- 172.21.15.78:0/2467915932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c419d480 con 0x7f33c4103660 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.681+0000 7f33ca024640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 0x7f33c41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33c9823640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d950 tx=0x7f33b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33bb7fe640 1 -- 172.21.15.78:0/2467915932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4017030 con 0x7f33c4103660 2024-09-16T01:15:10.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c419d770 con 0x7f33c4103660 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33bb7fe640 1 -- 172.21.15.78:0/2467915932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33b4004020 con 0x7f33c4103660 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33bb7fe640 1 -- 172.21.15.78:0/2467915932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b40047a0 con 0x7f33c4103660 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.682+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c41a5050 con 0x7f33c4103660 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.684+0000 7f33bb7fe640 1 -- 172.21.15.78:0/2467915932 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f33b40041d0 con 0x7f33c4103660 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.684+0000 7f33ce661740 1 Processor -- start 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 -- start start 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33bb7fe640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 0x7f33b00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f120d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f123630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55924f07b620 con 0x55924f116060 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55924f022e50 con 0x55924f116980 2024-09-16T01:15:10.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ce661740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55924f0775e0 con 0x55924f121240 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca024640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 0x7f33b00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37626/0 (socket says 172.21.15.78:37626) 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 -- 172.21.15.78:0/365033193 learned_addr learned my addr 172.21.15.78:0/365033193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33bb7fe640 1 -- 172.21.15.78:0/2467915932 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33b40cfad0 con 0x7f33c4103660 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33c9823640 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f120d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f123630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 -- 172.21.15.78:0/365033193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 msgr2=0x55924f123630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f123630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 -- 172.21.15.78:0/365033193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 msgr2=0x55924f120d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f120d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 -- 172.21.15.78:0/365033193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924f013d40 con 0x55924f116060 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.685+0000 7f33ca825640 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007f00 tx=0x7f33bc0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6283166e0e607317 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ca024640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 0x7f33b00a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33c0005ab0 tx=0x7f33c0005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33b8dea640 1 -- 172.21.15.78:0/365033193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc018030 con 0x55924f116060 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33b8dea640 1 -- 172.21.15.78:0/365033193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33bc0107c0 con 0x55924f116060 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33b8dea640 1 -- 172.21.15.78:0/365033193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc005370 con 0x55924f116060 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 -- 172.21.15.78:0/365033193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 msgr2=0x55924f116440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007f00 tx=0x7f33bc0077d0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 -- 172.21.15.78:0/365033193 shutdown_connections 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f123630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f120d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 --2- 172.21.15.78:0/365033193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f116440 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 -- 172.21.15.78:0/365033193 >> 172.21.15.78:0/365033193 conn(0x55924f1125f0 msgr2=0x55924f1129d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.686+0000 7f33ce661740 1 -- 172.21.15.78:0/365033193 shutdown_connections 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 -- 172.21.15.78:0/365033193 wait complete. 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 -- start start 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 Processor -- start 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116060 0x55924f11e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f11ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55924f123e10 con 0x55924f116980 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55924f003f10 con 0x55924f116060 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55924f123f90 con 0x55924f121240 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ce661740 10 client.0 _ll_register_callbacks cb 0x55924f027db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33c9823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33c9823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37642/0 (socket says 172.21.15.78:37642) 2024-09-16T01:15:10.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33c9823640 1 -- 172.21.15.78:0/304579285 learned_addr learned my addr 172.21.15.78:0/304579285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ca024640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116060 0x55924f11e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33ca825640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f11ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33c9823640 1 -- 172.21.15.78:0/304579285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 msgr2=0x55924f11ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.687+0000 7f33c9823640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f11ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33c9823640 1 -- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116060 msgr2=0x55924f11e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33c9823640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116060 0x55924f11e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33c9823640 1 -- 172.21.15.78:0/304579285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924ed8a580 con 0x55924f116980 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33c9823640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f33b409b720 tx=0x7f33b409b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709ed1fdd35f6930 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b409c120 con 0x55924f116980 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33b40cd110 con 0x55924f116980 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33bc0107c0 con 0x55924f116980 2024-09-16T01:15:10.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33b40ae330 con 0x55924f116980 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 10 client.15918 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55924ee54960 con 0x55924f116980 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33bc0107c0 con 0x55924f116980 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33b4098090 con 0x55924f116980 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 10 client.15918 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 10 client.15918 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 20 client.15918 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f130d60 0x55924f1331d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 4 client.15918 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.688+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55924f088b90 con 0x55924f130d60 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.689+0000 7f33ca024640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f130d60 0x55924f1331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.689+0000 7f33adffb640 10 client.15918 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:10.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.689+0000 7f33ca024640 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f130d60 0x55924f1331d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e04b4f7adf74611d server_cookie=47356a23eac26a0c in_seq=0 out_seq=0 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33adffb640 1 -- 172.21.15.78:0/304579285 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55924f088b90 con 0x55924f130d60 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33adffb640 10 client.15918 handle_command_reply: tid=0 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 client.15918 shutdown 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 msgr2=0x55924f11e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f33b409b720 tx=0x7f33b409b440 comp rx=0 tx=0).stop 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 shutdown_connections 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f130d60 0x55924f1331d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f11ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116980 0x55924f11e6d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 --2- 172.21.15.78:0/304579285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116060 0x55924f11e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 >> 172.21.15.78:0/304579285 conn(0x55924f1125f0 msgr2=0x55924f118320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33adffb640 0 client.15918 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 shutdown_connections 2024-09-16T01:15:10.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 1 -- 172.21.15.78:0/304579285 wait complete. 2024-09-16T01:15:10.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.690+0000 7f33ce661740 20 client.15918 trim_cache size 0 max 16384 2024-09-16T01:15:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.707+0000 7f33ce661740 1 Processor -- start 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.707+0000 7f33ce661740 1 -- start start 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f116980 0x55924f138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 0x55924f138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41b1960 con 0x55924f116060 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41af230 con 0x55924f121240 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ce661740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41ad080 con 0x55924f116980 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37646/0 (socket says 172.21.15.78:37646) 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 -- 172.21.15.78:0/569399296 learned_addr learned my addr 172.21.15.78:0/569399296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33c9823640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f116980 0x55924f138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 -- 172.21.15.78:0/569399296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f116980 msgr2=0x55924f138aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca825640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 0x55924f138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f116980 0x55924f138aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 -- 172.21.15.78:0/569399296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 msgr2=0x55924f138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 0x55924f138fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 -- 172.21.15.78:0/569399296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924f110bb0 con 0x55924f116060 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca825640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 0x55924f138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33ca024640 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009900 tx=0x7f33c003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa5984390cc50d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33affff640 1 -- 172.21.15.78:0/569399296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0035050 con 0x55924f116060 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33affff640 1 -- 172.21.15.78:0/569399296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33c00319f0 con 0x55924f116060 2024-09-16T01:15:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.708+0000 7f33affff640 1 -- 172.21.15.78:0/569399296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c00035f0 con 0x55924f116060 2024-09-16T01:15:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 -- 172.21.15.78:0/569399296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 msgr2=0x55924f111640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009900 tx=0x7f33c003c000 comp rx=0 tx=0).stop 2024-09-16T01:15:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 -- 172.21.15.78:0/569399296 shutdown_connections 2024-09-16T01:15:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f121240 0x55924f138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f116980 0x55924f138aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 --2- 172.21.15.78:0/569399296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f111640 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 -- 172.21.15.78:0/569399296 >> 172.21.15.78:0/569399296 conn(0x55924f1125f0 msgr2=0x55924f110f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 -- 172.21.15.78:0/569399296 shutdown_connections 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.709+0000 7f33ce661740 1 -- 172.21.15.78:0/569399296 wait complete. 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 -- start start 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 Processor -- start 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f129900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f1261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41ad6f0 con 0x55924f116060 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41aeb60 con 0x55924f116980 2024-09-16T01:15:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41addc0 con 0x55924f121240 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ce661740 10 client.0 _ll_register_callbacks cb 0x7f33b409b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37660/0 (socket says 172.21.15.78:37660) 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 -- 172.21.15.78:0/2100396130 learned_addr learned my addr 172.21.15.78:0/2100396130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33c9823640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f129900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca825640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f1261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 -- 172.21.15.78:0/2100396130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 msgr2=0x55924f1261b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f1261b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 -- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 msgr2=0x55924f129900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f129900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33ca024640 1 -- 172.21.15.78:0/2100396130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924f126990 con 0x55924f116060 2024-09-16T01:15:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.710+0000 7f33c9823640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f129900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ca024640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f33c0004ef0 tx=0x7f33c0004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5623ed2948c02974 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c005a070 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33c0005240 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33c00319f0 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33c0066050 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 10 client.15924 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55924f19c510 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33c00319f0 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33c0052450 con 0x55924f116060 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 10 client.15924 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 10 client.15924 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 20 client.15924 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f19cc40 0x55924f19f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 4 client.15924 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1b5f538a", "release": true, "await": true}] 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.711+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1b5f538a", "release": true, "await": true}) -- 0x55924f0830a0 con 0x55924f19cc40 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.712+0000 7f33c9823640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f19cc40 0x55924f19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.712+0000 7f33adffb640 10 client.15924 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.712+0000 7f33c9823640 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f19cc40 0x55924f19f060 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2c910db11e34ce server_cookie=1690c05898453be1 in_seq=0 out_seq=0 2024-09-16T01:15:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33adffb640 1 -- 172.21.15.78:0/2100396130 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55924f0830a0 con 0x55924f19cc40 2024-09-16T01:15:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33adffb640 10 client.15924 handle_command_reply: tid=0 2024-09-16T01:15:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 client.15924 shutdown 2024-09-16T01:15:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 msgr2=0x55924f1293c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f33c0004ef0 tx=0x7f33c0004f20 comp rx=0 tx=0).stop 2024-09-16T01:15:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 shutdown_connections 2024-09-16T01:15:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55924f19cc40 0x55924f19f060 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55924f121240 0x55924f1261b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55924f116980 0x55924f129900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.760+0000 7f33ce661740 1 --2- 172.21.15.78:0/2100396130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55924f116060 0x55924f1293c0 secure :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f33c0004ef0 tx=0x7f33c0004f20 comp rx=0 tx=0).stop 2024-09-16T01:15:10.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.761+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 >> 172.21.15.78:0/2100396130 conn(0x55924f1125f0 msgr2=0x55924f11e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.761+0000 7f33adffb640 0 client.15924 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.771+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 shutdown_connections 2024-09-16T01:15:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.771+0000 7f33ce661740 1 -- 172.21.15.78:0/2100396130 wait complete. 2024-09-16T01:15:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.771+0000 7f33ce661740 20 client.15924 trim_cache size 0 max 16384 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 16, 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "1b5f538a": { 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 16, 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:15:10.773 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 133.8, 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:10.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:10.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:10.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:10.775 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 msgr2=0x7f33b00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 0x7f33b00a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33c0005ab0 tx=0x7f33c0005a20 comp rx=0 tx=0).stop 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 msgr2=0x7f33c4198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f33b400d950 tx=0x7f33b400de10 comp rx=0 tx=0).stop 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 shutdown_connections 2024-09-16T01:15:10.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33b00a33b0 0x7f33b00a5870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c410c170 0x7f33c419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33c4103660 0x7f33c4198a10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 --2- 172.21.15.78:0/2467915932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c4102d40 0x7f33c41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.772+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 >> 172.21.15.78:0/2467915932 conn(0x7f33c40fc7a0 msgr2=0x7f33c40fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.773+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 shutdown_connections 2024-09-16T01:15:10.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:10.773+0000 7f33cc288640 1 -- 172.21.15.78:0/2467915932 wait complete. 2024-09-16T01:15:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 235.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:15:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:11 smithi078 ceph-mon[30937]: pgmap v134: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-09-16T01:15:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:11 smithi193 ceph-mon[35701]: pgmap v134: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-09-16T01:15:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:11 smithi148 ceph-mon[35030]: pgmap v134: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.61k op/s 2024-09-16T01:15:11.672 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1b5f538a', total seconds elapsed: 17.4 2024-09-16T01:15:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:13 smithi078 ceph-mon[30937]: pgmap v135: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:13 smithi193 ceph-mon[35701]: pgmap v135: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:13.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:13 smithi148 ceph-mon[35030]: pgmap v135: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:14.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 -- 172.21.15.78:0/127890014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf10101370 msgr2=0x7fdf1006a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 --2- 172.21.15.78:0/127890014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf10101370 0x7fdf1006a0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009ca0 tx=0x7fdf0802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 -- 172.21.15.78:0/127890014 shutdown_connections 2024-09-16T01:15:14.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 --2- 172.21.15.78:0/127890014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf10101370 0x7fdf1006a0c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 --2- 172.21.15.78:0/127890014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf101009d0 0x7fdf10100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 --2- 172.21.15.78:0/127890014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 0x7fdf10106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.829+0000 7fdf1760a640 1 -- 172.21.15.78:0/127890014 >> 172.21.15.78:0/127890014 conn(0x7fdf100fc6b0 msgr2=0x7fdf100fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.831+0000 7fdf1760a640 1 -- 172.21.15.78:0/127890014 shutdown_connections 2024-09-16T01:15:14.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.831+0000 7fdf1760a640 1 -- 172.21.15.78:0/127890014 wait complete. 2024-09-16T01:15:14.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.831+0000 7fdf1760a640 1 Processor -- start 2024-09-16T01:15:14.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 -- start start 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf10101370 0x7fdf10111790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 0x7fdf10111cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1010d550 con 0x7fdf10101370 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1010d3d0 con 0x7fdf101009d0 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf1760a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1010d6d0 con 0x7fdf101069e0 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:40806/0 (socket says 172.21.15.78:40806) 2024-09-16T01:15:14.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 -- 172.21.15.78:0/3979463152 learned_addr learned my addr 172.21.15.78:0/3979463152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 0x7fdf10111cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf10101370 0x7fdf10111790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 -- 172.21.15.78:0/3979463152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 msgr2=0x7fdf10111cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 0x7fdf10111cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 -- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf10101370 msgr2=0x7fdf10111790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf10101370 0x7fdf10111790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.832+0000 7fdf153a6640 1 -- 172.21.15.78:0/3979463152 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf101aabb0 con 0x7fdf101009d0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf153a6640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00d950 tx=0x7fdf0c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf027fc640 1 -- 172.21.15.78:0/3979463152 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c017030 con 0x7fdf101009d0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf027fc640 1 -- 172.21.15.78:0/3979463152 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf0c0099b0 con 0x7fdf101009d0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf101aaea0 con 0x7fdf101009d0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf027fc640 1 -- 172.21.15.78:0/3979463152 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0047a0 con 0x7fdf101009d0 2024-09-16T01:15:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.833+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf101ab350 con 0x7fdf101009d0 2024-09-16T01:15:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.836+0000 7fdf027fc640 1 -- 172.21.15.78:0/3979463152 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fdf0c004020 con 0x7fdf101009d0 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.837+0000 7fdf1a477740 1 Processor -- start 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.837+0000 7fdf027fc640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 0x7fdefc0a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.837+0000 7fdf1a477740 1 -- start start 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf027fc640 1 -- 172.21.15.78:0/3979463152 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdf0c0cf890 con 0x7fdf101009d0 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916e3410 0x55e4916ed770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916edcb0 0x55e4916f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e4916472f0 con 0x55e4916e3410 2024-09-16T01:15:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e4915ef270 con 0x55e4916e2af0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf1a477740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e4916432b0 con 0x55e4916edcb0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 0x7fdefc0a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:40808/0 (socket says 172.21.15.78:40808) 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 -- 172.21.15.78:0/1589439582 learned_addr learned my addr 172.21.15.78:0/1589439582 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916e3410 0x55e4916ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 -- 172.21.15.78:0/1589439582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916edcb0 msgr2=0x55e4916f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916edcb0 0x55e4916f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 -- 172.21.15.78:0/1589439582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916e3410 msgr2=0x55e4916ed770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916e3410 0x55e4916ed770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 -- 172.21.15.78:0/1589439582 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4915ee340 con 0x55e4916e2af0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 0x7fdefc0a59f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdf08004ab0 tx=0x7fdf080098c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.838+0000 7fdf153a6640 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0a3f20 tx=0x7fdf0c09b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=972eba591cdfcf58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdefb7fe640 1 -- 172.21.15.78:0/1589439582 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c09b7e0 con 0x55e4916e2af0 2024-09-16T01:15:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdefb7fe640 1 -- 172.21.15.78:0/1589439582 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf0c09fb90 con 0x55e4916e2af0 2024-09-16T01:15:14.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdefb7fe640 1 -- 172.21.15.78:0/1589439582 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c009b60 con 0x55e4916e2af0 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 -- 172.21.15.78:0/1589439582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 msgr2=0x55e4916e2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0a3f20 tx=0x7fdf0c09b670 comp rx=0 tx=0).stop 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 -- 172.21.15.78:0/1589439582 shutdown_connections 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916edcb0 0x55e4916f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916e3410 0x55e4916ed770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 --2- 172.21.15.78:0/1589439582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916e2ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 -- 172.21.15.78:0/1589439582 >> 172.21.15.78:0/1589439582 conn(0x55e4916df060 msgr2=0x55e4916df440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 -- 172.21.15.78:0/1589439582 shutdown_connections 2024-09-16T01:15:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.839+0000 7fdf1a477740 1 -- 172.21.15.78:0/1589439582 wait complete. 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 -- start start 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 Processor -- start 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e3410 0x55e4916eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916eb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e4916f0880 con 0x55e4916edcb0 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e4915d0290 con 0x55e4916e2af0 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e4916f0a00 con 0x55e4916e3410 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf1a477740 10 client.0 _ll_register_callbacks cb 0x55e491601140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf153a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:40816/0 (socket says 172.21.15.78:40816) 2024-09-16T01:15:14.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf153a6640 1 -- 172.21.15.78:0/2003638769 learned_addr learned my addr 172.21.15.78:0/2003638769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.840+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916eb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e3410 0x55e4916eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 -- 172.21.15.78:0/2003638769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e3410 msgr2=0x55e4916eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e3410 0x55e4916eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 -- 172.21.15.78:0/2003638769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 msgr2=0x55e4916eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916eb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 -- 172.21.15.78:0/2003638769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e49135a210 con 0x55e4916e2af0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf153a6640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0ccd00 tx=0x7fdf0c0cced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534a92e1bcb9775d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdef8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0b8020 con 0x55e4916e2af0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdf0c097e50 con 0x55e4916e2af0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf0c09fb90 con 0x55e4916e2af0 2024-09-16T01:15:14.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c083050 con 0x55e4916e2af0 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf1a477740 10 client.25385 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.841+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e491426620 con 0x55e4916e2af0 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf0c09b7e0 con 0x55e4916e2af0 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdf0c084ae0 con 0x55e4916e2af0 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 10 client.25385 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 10 client.25385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 20 client.25385 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e4916fff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 4 client.25385 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdf0c09fb90 con 0x55e4916fdb20 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.842+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e4916fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.843+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e4916fff10 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606175693b425675 server_cookie=d29ac91915a954be in_seq=0 out_seq=0 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.843+0000 7fdef8ff9640 10 client.25385 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.844+0000 7fdef8ff9640 1 -- 172.21.15.78:0/2003638769 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fdf0c09fb90 con 0x55e4916fdb20 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.844+0000 7fdef8ff9640 10 client.25385 handle_command_reply: tid=0 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.844+0000 7fdf1a477740 1 client.25385 shutdown 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 msgr2=0x55e4916eac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0ccd00 tx=0x7fdf0c0cced0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 shutdown_connections 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e4916fff10 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdef8ff9640 0 client.25385 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e3410 0x55e4916eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 --2- 172.21.15.78:0/2003638769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e2af0 0x55e4916eac40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 >> 172.21.15.78:0/2003638769 conn(0x55e4916df060 msgr2=0x55e4916e4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 shutdown_connections 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.845+0000 7fdf1a477740 1 -- 172.21.15.78:0/2003638769 wait complete. 2024-09-16T01:15:14.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.846+0000 7fdf1a477740 20 client.25385 trim_cache size 0 max 16384 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 Processor -- start 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 -- start start 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b7ea0 con 0x55e4916edcb0 2024-09-16T01:15:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b5770 con 0x55e4916e3410 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.865+0000 7fdf1a477740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b35c0 con 0x55e4916e2af0 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf14ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49628/0 (socket says 172.21.15.78:49628) 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 -- 172.21.15.78:0/598342469 learned_addr learned my addr 172.21.15.78:0/598342469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf153a6640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 -- 172.21.15.78:0/598342469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 msgr2=0x55e4916eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 -- 172.21.15.78:0/598342469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 msgr2=0x55e4916ecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916ecbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 -- 172.21.15.78:0/598342469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4916e2920 con 0x55e4916edcb0 2024-09-16T01:15:14.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf153a6640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdf08035000 tx=0x7fdf08041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebddb0f49fa60840 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdefaffd640 1 -- 172.21.15.78:0/598342469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08042070 con 0x55e4916edcb0 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdefaffd640 1 -- 172.21.15.78:0/598342469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf08044070 con 0x55e4916edcb0 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.866+0000 7fdefaffd640 1 -- 172.21.15.78:0/598342469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0802f870 con 0x55e4916edcb0 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.867+0000 7fdf1a477740 1 -- 172.21.15.78:0/598342469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 msgr2=0x55e491705510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.867+0000 7fdf1a477740 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fdf08035000 tx=0x7fdf08041000 comp rx=0 tx=0).stop 2024-09-16T01:15:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.868+0000 7fdf1a477740 1 -- 172.21.15.78:0/598342469 shutdown_connections 2024-09-16T01:15:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.868+0000 7fdf1a477740 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e491705510 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.868+0000 7fdf1a477740 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.868+0000 7fdf1a477740 1 --2- 172.21.15.78:0/598342469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.868+0000 7fdf1a477740 1 -- 172.21.15.78:0/598342469 >> 172.21.15.78:0/598342469 conn(0x55e4916df060 msgr2=0x55e4916ec4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.869+0000 7fdf1a477740 1 -- 172.21.15.78:0/598342469 shutdown_connections 2024-09-16T01:15:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.869+0000 7fdf1a477740 1 -- 172.21.15.78:0/598342469 wait complete. 2024-09-16T01:15:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.870+0000 7fdf1a477740 1 -- start start 2024-09-16T01:15:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.870+0000 7fdf1a477740 1 Processor -- start 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.870+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.870+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916e7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.870+0000 7fdf1a477740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf1a477740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b3c30 con 0x55e4916edcb0 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf14ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916e7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf14ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916e7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:40824/0 (socket says 172.21.15.78:40824) 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf14ba5640 1 -- 172.21.15.78:0/582545281 learned_addr learned my addr 172.21.15.78:0/582545281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf15ba7640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b50a0 con 0x55e4916e3410 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf101b4300 con 0x55e4916e2af0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf1a477740 10 client.0 _ll_register_callbacks cb 0x7fdf040097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 -- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 msgr2=0x55e4916e7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916e7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 -- 172.21.15.78:0/582545281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 msgr2=0x55e4916e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4916e8640 con 0x55e4916e2af0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdf153a6640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916f63b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c097c40 tx=0x7fdf0c098080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f07d7b37ba7a30e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.871+0000 7fdef8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c09b140 con 0x55e4916e2af0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdf0c09ac30 con 0x55e4916e2af0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf08044070 con 0x55e4916e2af0 2024-09-16T01:15:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c097a80 con 0x55e4916e2af0 2024-09-16T01:15:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 10 client.25438 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e491768460 con 0x55e4916e2af0 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf08044070 con 0x55e4916e2af0 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdf0c084ae0 con 0x55e4916e2af0 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 10 client.25438 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 10 client.25438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 20 client.25438 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e49176a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 4 client.25438 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.872+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e491654a20 con 0x55e4916fdb20 2024-09-16T01:15:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.873+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e49176a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.873+0000 7fdef8ff9640 10 client.25438 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.873+0000 7fdf14ba5640 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e49176a9f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ab20fc9b58f068e server_cookie=e7e8eeef3b9d1e1a in_seq=0 out_seq=0 2024-09-16T01:15:14.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdef8ff9640 1 -- 172.21.15.78:0/582545281 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e491654a20 con 0x55e4916fdb20 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdef8ff9640 10 client.25438 handle_command_reply: tid=0 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 client.25438 shutdown 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 msgr2=0x55e4916f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916f63b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c097c40 tx=0x7fdf0c098080 comp rx=0 tx=0).stop 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 shutdown_connections 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e4916fdb20 0x55e49176a9f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e4916edcb0 0x55e4916e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e4916e3410 0x55e4916e7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 --2- 172.21.15.78:0/582545281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4916e2af0 0x55e4916f63b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 >> 172.21.15.78:0/582545281 conn(0x55e4916df060 msgr2=0x55e4916e4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdef8ff9640 0 client.25438 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 shutdown_connections 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.882+0000 7fdf1a477740 1 -- 172.21.15.78:0/582545281 wait complete. 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1a477740 20 client.25438 trim_cache size 0 max 16384 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:15:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 msgr2=0x7fdefc0a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 0x7fdefc0a59f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdf08004ab0 tx=0x7fdf080098c0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 msgr2=0x7fdf10111250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00d950 tx=0x7fdf0c00de10 comp rx=0 tx=0).stop 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 shutdown_connections 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdefc0a3530 0x7fdefc0a59f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf101069e0 0x7fdf10111cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf10101370 0x7fdf10111790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 --2- 172.21.15.78:0/3979463152 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf101009d0 0x7fdf10111250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 >> 172.21.15.78:0/3979463152 conn(0x7fdf100fc6b0 msgr2=0x7fdf1010daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.883+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 shutdown_connections 2024-09-16T01:15:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:14.884+0000 7fdf1760a640 1 -- 172.21.15.78:0/3979463152 wait complete. 2024-09-16T01:15:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:15 smithi078 ceph-mon[30937]: pgmap v136: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:15 smithi193 ceph-mon[35701]: pgmap v136: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:15.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:15 smithi148 ceph-mon[35030]: pgmap v136: 129 pgs: 129 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 1.02k op/s 2024-09-16T01:15:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 240.0 s, 1406.1 tps, lat 201.158 ms stddev 1142.667 2024-09-16T01:15:17.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:17 smithi078 ceph-mon[30937]: pgmap v137: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:17 smithi193 ceph-mon[35701]: pgmap v137: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:17.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:17 smithi148 ceph-mon[35030]: pgmap v137: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:19.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:19 smithi078 ceph-mon[30937]: pgmap v138: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-09-16T01:15:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:19 smithi193 ceph-mon[35701]: pgmap v138: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-09-16T01:15:19.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:19 smithi148 ceph-mon[35030]: pgmap v138: 129 pgs: 129 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.8 MiB/s wr, 285 op/s 2024-09-16T01:15:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 245.0 s, 1903.2 tps, lat 52.246 ms stddev 65.249 2024-09-16T01:15:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:21 smithi078 ceph-mon[30937]: pgmap v139: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.1 MiB/s wr, 544 op/s 2024-09-16T01:15:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:21 smithi193 ceph-mon[35701]: pgmap v139: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.1 MiB/s wr, 544 op/s 2024-09-16T01:15:21.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:21 smithi148 ceph-mon[35030]: pgmap v139: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 7.1 MiB/s wr, 544 op/s 2024-09-16T01:15:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:23 smithi078 ceph-mon[30937]: pgmap v140: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 5.0 MiB/s wr, 412 op/s 2024-09-16T01:15:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:23 smithi193 ceph-mon[35701]: pgmap v140: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 5.0 MiB/s wr, 412 op/s 2024-09-16T01:15:23.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:23 smithi148 ceph-mon[35030]: pgmap v140: 129 pgs: 129 active+clean; 2.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 5.0 MiB/s wr, 412 op/s 2024-09-16T01:15:25.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:25 smithi078 ceph-mon[30937]: pgmap v141: 129 pgs: 129 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.0 MiB/s wr, 417 op/s 2024-09-16T01:15:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:25 smithi193 ceph-mon[35701]: pgmap v141: 129 pgs: 129 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.0 MiB/s wr, 417 op/s 2024-09-16T01:15:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:25 smithi148 ceph-mon[35030]: pgmap v141: 129 pgs: 129 active+clean; 2.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 5.0 MiB/s wr, 417 op/s 2024-09-16T01:15:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 250.0 s, 1617.4 tps, lat 62.274 ms stddev 70.952 2024-09-16T01:15:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:27 smithi078 ceph-mon[30937]: pgmap v142: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.3 MiB/s wr, 663 op/s 2024-09-16T01:15:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:27 smithi193 ceph-mon[35701]: pgmap v142: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.3 MiB/s wr, 663 op/s 2024-09-16T01:15:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:27 smithi148 ceph-mon[35030]: pgmap v142: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.3 MiB/s wr, 663 op/s 2024-09-16T01:15:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/2918175700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb8410c0d0 msgr2=0x7ffb8410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/2918175700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb8410c0d0 0x7ffb8410e570 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009ca0 tx=0x7ffb7c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/2918175700 shutdown_connections 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/2918175700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb8410c0d0 0x7ffb8410e570 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/2918175700 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb841035c0 0x7ffb8410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/2918175700 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb84102ca0 0x7ffb84103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.007+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/2918175700 >> 172.21.15.78:0/2918175700 conn(0x7ffb840fc6b0 msgr2=0x7ffb840fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.008+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/2918175700 shutdown_connections 2024-09-16T01:15:28.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.008+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/2918175700 wait complete. 2024-09-16T01:15:28.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.009+0000 7ffb8a2e6640 1 Processor -- start 2024-09-16T01:15:28.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.009+0000 7ffb8a2e6640 1 -- start start 2024-09-16T01:15:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb841035c0 0x7ffb841986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb8410c0d0 0x7ffb8419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84111f00 con 0x7ffb841035c0 2024-09-16T01:15:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84111d80 con 0x7ffb8410c0d0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb8a2e6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84112080 con 0x7ffb84102ca0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35886/0 (socket says 172.21.15.78:35886) 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb83fff640 1 -- 172.21.15.78:0/3620951194 learned_addr learned my addr 172.21.15.78:0/3620951194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb837fe640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb841035c0 0x7ffb841986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.010+0000 7ffb88883640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb8410c0d0 0x7ffb8419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 -- 172.21.15.78:0/3620951194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb8410c0d0 msgr2=0x7ffb8419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb8410c0d0 0x7ffb8419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 -- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb841035c0 msgr2=0x7ffb841986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb841035c0 0x7ffb841986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 -- 172.21.15.78:0/3620951194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8419d180 con 0x7ffb84102ca0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb83fff640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400ea60 tx=0x7ffb7400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb817fa640 1 -- 172.21.15.78:0/3620951194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7400cde0 con 0x7ffb84102ca0 2024-09-16T01:15:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8419d470 con 0x7ffb84102ca0 2024-09-16T01:15:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb817fa640 1 -- 172.21.15.78:0/3620951194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb740035f0 con 0x7ffb84102ca0 2024-09-16T01:15:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb817fa640 1 -- 172.21.15.78:0/3620951194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74010670 con 0x7ffb84102ca0 2024-09-16T01:15:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.011+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb841a4d00 con 0x7ffb84102ca0 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.014+0000 7ffb8d076740 1 Processor -- start 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb817fa640 1 -- 172.21.15.78:0/3620951194 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7ffb740037a0 con 0x7ffb84102ca0 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 -- start start 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c69000 0x562e32c749c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562e32bc32f0 con 0x562e32c69000 2024-09-16T01:15:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562e32b6b270 con 0x562e32c6ccc0 2024-09-16T01:15:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb8d076740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e32bbf2b0 con 0x562e32c68660 2024-09-16T01:15:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c69000 0x562e32c749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb88883640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb88883640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50176/0 (socket says 172.21.15.78:50176) 2024-09-16T01:15:28.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb88883640 1 -- 172.21.15.78:0/2285400412 learned_addr learned my addr 172.21.15.78:0/2285400412 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb817fa640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 0x7ffb6c0a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.015+0000 7ffb83fff640 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 -- 172.21.15.78:0/2285400412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 msgr2=0x562e32c68ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c68ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 -- 172.21.15.78:0/2285400412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c69000 msgr2=0x562e32c749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c69000 0x562e32c749c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 -- 172.21.15.78:0/2285400412 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e32b6a340 con 0x562e32c6ccc0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb817fa640 1 -- 172.21.15.78:0/3620951194 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffb740d0360 con 0x7ffb84102ca0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 0x7ffb6c0a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009ca0 tx=0x7ffb7c001040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b04e8ba72965ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb88883640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 0x7ffb6c0a5a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c04d000 tx=0x7ffb7c04e000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb6a7fc640 1 -- 172.21.15.78:0/2285400412 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c042040 con 0x562e32c6ccc0 2024-09-16T01:15:28.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb6a7fc640 1 -- 172.21.15.78:0/2285400412 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb7c036070 con 0x562e32c6ccc0 2024-09-16T01:15:28.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.016+0000 7ffb6a7fc640 1 -- 172.21.15.78:0/2285400412 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c004740 con 0x562e32c6ccc0 2024-09-16T01:15:28.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.017+0000 7ffb8d076740 1 -- 172.21.15.78:0/2285400412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 msgr2=0x562e32c6d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.017+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009ca0 tx=0x7ffb7c001040 comp rx=0 tx=0).stop 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.018+0000 7ffb8d076740 1 -- 172.21.15.78:0/2285400412 shutdown_connections 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.018+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c69000 0x562e32c749c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.018+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c68ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.018+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2285400412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c6ccc0 0x562e32c6d0a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.018+0000 7ffb8d076740 1 -- 172.21.15.78:0/2285400412 >> 172.21.15.78:0/2285400412 conn(0x562e32c5ab10 msgr2=0x562e32c5aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.019+0000 7ffb8d076740 1 -- 172.21.15.78:0/2285400412 shutdown_connections 2024-09-16T01:15:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.019+0000 7ffb8d076740 1 -- 172.21.15.78:0/2285400412 wait complete. 2024-09-16T01:15:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.019+0000 7ffb8d076740 1 -- start start 2024-09-16T01:15:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 Processor -- start 2024-09-16T01:15:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c71e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562e32c750b0 con 0x562e32c6ccc0 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562e32b4c290 con 0x562e32c68660 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e32c75230 con 0x562e32c69000 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35890/0 (socket says 172.21.15.78:35890) 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb837fe640 1 -- 172.21.15.78:0/2357654566 learned_addr learned my addr 172.21.15.78:0/2357654566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.020+0000 7ffb8d076740 10 client.0 _ll_register_callbacks cb 0x562e32b7d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:28.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb83fff640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c71e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb837fe640 1 -- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 msgr2=0x562e32c71e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb837fe640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c71e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb837fe640 1 -- 172.21.15.78:0/2357654566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 msgr2=0x562e32c728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb837fe640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.021+0000 7ffb837fe640 1 -- 172.21.15.78:0/2357654566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e328d6210 con 0x562e32c69000 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.024+0000 7ffb837fe640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb78012700 tx=0x7ffb78012b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da8a56e4f677bdda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.025+0000 7ffb637fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.033+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78004050 con 0x562e32c69000 2024-09-16T01:15:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.033+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb7c036070 con 0x562e32c69000 2024-09-16T01:15:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.033+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffb780041f0 con 0x562e32c69000 2024-09-16T01:15:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.034+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ffb78025050 con 0x562e32c69000 2024-09-16T01:15:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.035+0000 7ffb8d076740 10 client.25450 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.035+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562e329a2620 con 0x562e32c69000 2024-09-16T01:15:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb78004430 con 0x562e32c69000 2024-09-16T01:15:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7ffb78003a40 con 0x562e32c69000 2024-09-16T01:15:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 10 client.25450 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 10 client.25450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 20 client.25450 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32c82950 0x562e32c84db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 4 client.25450 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.038+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffb7c036070 con 0x562e32c82950 2024-09-16T01:15:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.040+0000 7ffb83fff640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32c82950 0x562e32c84db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.045+0000 7ffb83fff640 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32c82950 0x562e32c84db0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e13dc122b5b234a6 server_cookie=51c4ba63ce406761 in_seq=0 out_seq=0 2024-09-16T01:15:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.046+0000 7ffb637fe640 10 client.25450 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.048+0000 7ffb637fe640 1 -- 172.21.15.78:0/2357654566 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7ffb7c036070 con 0x562e32c82950 2024-09-16T01:15:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.048+0000 7ffb637fe640 10 client.25450 handle_command_reply: tid=0 2024-09-16T01:15:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.048+0000 7ffb8d076740 1 client.25450 shutdown 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.053+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 msgr2=0x562e32c723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.053+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb78012700 tx=0x7ffb78012b00 comp rx=0 tx=0).stop 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 shutdown_connections 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32c82950 0x562e32c84db0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c69000 0x562e32c723b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 --2- 172.21.15.78:0/2357654566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c71e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 >> 172.21.15.78:0/2357654566 conn(0x562e32c5ab10 msgr2=0x562e32c67e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.054+0000 7ffb637fe640 0 client.25450 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.056+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 shutdown_connections 2024-09-16T01:15:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.056+0000 7ffb8d076740 1 -- 172.21.15.78:0/2357654566 wait complete. 2024-09-16T01:15:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.059+0000 7ffb8d076740 20 client.25450 trim_cache size 0 max 16384 2024-09-16T01:15:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.111+0000 7ffb8d076740 1 Processor -- start 2024-09-16T01:15:28.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.112+0000 7ffb8d076740 1 -- start start 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.112+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c72720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.113+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.113+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c8a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.113+0000 7ffb8d076740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841af770 con 0x562e32c6ccc0 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.113+0000 7ffb8d076740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841ad040 con 0x562e32c69000 2024-09-16T01:15:28.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.113+0000 7ffb8d076740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841aae90 con 0x562e32c68660 2024-09-16T01:15:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.114+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.114+0000 7ffb837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50194/0 (socket says 172.21.15.78:50194) 2024-09-16T01:15:28.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.114+0000 7ffb837fe640 1 -- 172.21.15.78:0/3126329435 learned_addr learned my addr 172.21.15.78:0/3126329435 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb83fff640 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c72720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 -- 172.21.15.78:0/3126329435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 msgr2=0x562e32c72720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c72720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 -- 172.21.15.78:0/3126329435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 msgr2=0x562e32c8a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c8a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 -- 172.21.15.78:0/3126329435 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e32c7ee90 con 0x562e32c69000 2024-09-16T01:15:28.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.115+0000 7ffb837fe640 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb78000c00 tx=0x7ffb78009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1fe751c097de173 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.118+0000 7ffb69ffb640 1 -- 172.21.15.78:0/3126329435 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7802f070 con 0x562e32c69000 2024-09-16T01:15:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.118+0000 7ffb69ffb640 1 -- 172.21.15.78:0/3126329435 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb78009e40 con 0x562e32c69000 2024-09-16T01:15:28.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.118+0000 7ffb69ffb640 1 -- 172.21.15.78:0/3126329435 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78010bc0 con 0x562e32c69000 2024-09-16T01:15:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 -- 172.21.15.78:0/3126329435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 msgr2=0x562e32c67d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb78000c00 tx=0x7ffb78009640 comp rx=0 tx=0).stop 2024-09-16T01:15:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 -- 172.21.15.78:0/3126329435 shutdown_connections 2024-09-16T01:15:28.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c8a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c69000 0x562e32c67d60 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb78000c00 tx=0x7ffb78009640 comp rx=0 tx=0).stop 2024-09-16T01:15:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 --2- 172.21.15.78:0/3126329435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c68660 0x562e32c72720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.121+0000 7ffb8d076740 1 -- 172.21.15.78:0/3126329435 >> 172.21.15.78:0/3126329435 conn(0x562e32c5ab10 msgr2=0x562e32c67650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.126 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.125+0000 7ffb8d076740 1 -- 172.21.15.78:0/3126329435 shutdown_connections 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.126+0000 7ffb8d076740 1 -- 172.21.15.78:0/3126329435 wait complete. 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.126+0000 7ffb8d076740 1 -- start start 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.126+0000 7ffb8d076740 1 Processor -- start 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c75eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 0x562e32c76870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841ab500 con 0x562e32c6ccc0 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841ac970 con 0x562e32c68660 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb841abbd0 con 0x562e32c763f0 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb8d076740 10 client.0 _ll_register_callbacks cb 0x7ffb78009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50204/0 (socket says 172.21.15.78:50204) 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 -- 172.21.15.78:0/4243746837 learned_addr learned my addr 172.21.15.78:0/4243746837 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb88883640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 0x562e32c76870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 -- 172.21.15.78:0/4243746837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 msgr2=0x562e32c76870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 0x562e32c76870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 -- 172.21.15.78:0/4243746837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 msgr2=0x562e32c75eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c75eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb83fff640 1 -- 172.21.15.78:0/4243746837 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e32cec840 con 0x562e32c68660 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.127+0000 7ffb88883640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 0x562e32c76870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.128+0000 7ffb83fff640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb740986b0 tx=0x7ffb740984f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66fc895fff4d87a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.128+0000 7ffb637fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:28.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb74003b30 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb78009e40 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7ffb740cd940 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ffb740988c0 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb8d076740 10 client.25409 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562e32cedb40 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.129+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb78009e40 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7ffb740bc0b0 con 0x562e32c68660 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 10 client.25409 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 10 client.25409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 20 client.25409 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32cee260 0x562e32cf06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 4 client.25409 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:15:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e32bd0a20 con 0x562e32cee260 2024-09-16T01:15:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.130+0000 7ffb837fe640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32cee260 0x562e32cf06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.131+0000 7ffb637fe640 10 client.25409 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.131+0000 7ffb837fe640 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32cee260 0x562e32cf06a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbaa147f47a9f90b server_cookie=39b169eef67eaa90 in_seq=0 out_seq=0 2024-09-16T01:15:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.140+0000 7ffb637fe640 1 -- 172.21.15.78:0/4243746837 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e32bd0a20 con 0x562e32cee260 2024-09-16T01:15:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.140+0000 7ffb637fe640 10 client.25409 handle_command_reply: tid=0 2024-09-16T01:15:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.140+0000 7ffb8d076740 1 client.25409 shutdown 2024-09-16T01:15:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 msgr2=0x562e32c7ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb740986b0 tx=0x7ffb740984f0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 shutdown_connections 2024-09-16T01:15:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562e32cee260 0x562e32cf06a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e32c763f0 0x562e32c76870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e32c6ccc0 0x562e32c75eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 --2- 172.21.15.78:0/4243746837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e32c68660 0x562e32c7ac30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 >> 172.21.15.78:0/4243746837 conn(0x562e32c5ab10 msgr2=0x562e32c67e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb637fe640 0 client.25409 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 shutdown_connections 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 1 -- 172.21.15.78:0/4243746837 wait complete. 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.142+0000 7ffb8d076740 20 client.25409 trim_cache size 0 max 16384 2024-09-16T01:15:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:15:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:15:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:15:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:15:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 msgr2=0x7ffb6c0a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 0x7ffb6c0a5a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c04d000 tx=0x7ffb7c04e000 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 msgr2=0x7ffb84198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400ea60 tx=0x7ffb7400ef20 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 shutdown_connections 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ffb6c0a3560 0x7ffb6c0a5a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb8410c0d0 0x7ffb8419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb841035c0 0x7ffb841986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 --2- 172.21.15.78:0/3620951194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb84102ca0 0x7ffb84198190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 >> 172.21.15.78:0/3620951194 conn(0x7ffb840fc6b0 msgr2=0x7ffb8410d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 shutdown_connections 2024-09-16T01:15:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:28.143+0000 7ffb8a2e6640 1 -- 172.21.15.78:0/3620951194 wait complete. 2024-09-16T01:15:29.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:29 smithi078 ceph-mon[30937]: pgmap v143: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-09-16T01:15:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:29 smithi193 ceph-mon[35701]: pgmap v143: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-09-16T01:15:29.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:29 smithi148 ceph-mon[35030]: pgmap v143: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-09-16T01:15:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 255.0 s, 1485.0 tps, lat 67.082 ms stddev 111.174 2024-09-16T01:15:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:31 smithi078 ceph-mon[30937]: pgmap v144: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-09-16T01:15:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:31 smithi193 ceph-mon[35701]: pgmap v144: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-09-16T01:15:31.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:31 smithi148 ceph-mon[35030]: pgmap v144: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 808 op/s 2024-09-16T01:15:33.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:33 smithi078 ceph-mon[30937]: pgmap v145: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 631 op/s 2024-09-16T01:15:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:33 smithi193 ceph-mon[35701]: pgmap v145: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 631 op/s 2024-09-16T01:15:33.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:33 smithi148 ceph-mon[35030]: pgmap v145: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.1 MiB/s wr, 631 op/s 2024-09-16T01:15:35.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:35 smithi078 ceph-mon[30937]: pgmap v146: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 637 op/s 2024-09-16T01:15:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:35 smithi193 ceph-mon[35701]: pgmap v146: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 637 op/s 2024-09-16T01:15:35.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:35 smithi148 ceph-mon[35030]: pgmap v146: 129 pgs: 129 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 637 op/s 2024-09-16T01:15:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 260.0 s, 1803.2 tps, lat 54.882 ms stddev 65.732 2024-09-16T01:15:36.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:36 smithi078 ceph-mon[30937]: pgmap v147: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-09-16T01:15:36.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:36 smithi193 ceph-mon[35701]: pgmap v147: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-09-16T01:15:36.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:36 smithi148 ceph-mon[35030]: pgmap v147: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-09-16T01:15:37.302 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T01:15:37.302 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-16T01:15:37.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:15:37.700 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 -- 172.21.15.78:0/1039851329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 msgr2=0x7f561810d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 --2- 172.21.15.78:0/1039851329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 0x7f561810d7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f561000f0d0 tx=0x7f561002a300 comp rx=0 tx=0).stop 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 -- 172.21.15.78:0/1039851329 shutdown_connections 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 --2- 172.21.15.78:0/1039851329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 0x7f561810d7a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 --2- 172.21.15.78:0/1039851329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181008d0 0x7f5618100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 --2- 172.21.15.78:0/1039851329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181068d0 0x7f5618106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 -- 172.21.15.78:0/1039851329 >> 172.21.15.78:0/1039851329 conn(0x7f56180fc610 msgr2=0x7f56180fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 -- 172.21.15.78:0/1039851329 shutdown_connections 2024-09-16T01:15:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.334+0000 7f561759e640 1 -- 172.21.15.78:0/1039851329 wait complete. 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.335+0000 7f561759e640 1 Processor -- start 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.335+0000 7f561759e640 1 -- start start 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.335+0000 7f561759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 0x7f56180739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.335+0000 7f561759e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 0x7f561806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 0x7f561806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561759e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f561810fba0 con 0x7f56181008d0 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f561810fa20 con 0x7f56181068d0 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561759e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f561810fd20 con 0x7f5618101270 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f5616d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 0x7f561806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f5616d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 0x7f561806cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36192/0 (socket says 172.21.15.78:36192) 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f5616d9d640 1 -- 172.21.15.78:0/4236486717 learned_addr learned my addr 172.21.15.78:0/4236486717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 0x7f56180739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 -- 172.21.15.78:0/4236486717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 msgr2=0x7f561806ca50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 0x7f561806ca50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 -- 172.21.15.78:0/4236486717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 msgr2=0x7f561806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 0x7f561806cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.336+0000 7f561659c640 1 -- 172.21.15.78:0/4236486717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561806d550 con 0x7f56181008d0 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f561659c640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 0x7f56180739c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f560800b3f0 tx=0x7f560800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608015020 con 0x7f56181008d0 2024-09-16T01:15:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5608011d10 con 0x7f56181008d0 2024-09-16T01:15:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561806d840 con 0x7f56181008d0 2024-09-16T01:15:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560800f2e0 con 0x7f56181008d0 2024-09-16T01:15:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.337+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5618071810 con 0x7f56181008d0 2024-09-16T01:15:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.338+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5618102010 con 0x7f56181008d0 2024-09-16T01:15:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.341+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f5608018020 con 0x7f56181008d0 2024-09-16T01:15:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.342+0000 7f560f7fe640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 0x7f56040a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.342+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f56080cfdb0 con 0x7f56181008d0 2024-09-16T01:15:38.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.343+0000 7f560ffff640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 0x7f56040a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:38.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.344+0000 7f560ffff640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 0x7f56040a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f561806e210 tx=0x7f560000fae0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.350+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f560801f020 con 0x7f56181008d0 2024-09-16T01:15:38.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:38 smithi148 ceph-mon[35030]: pgmap v148: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-09-16T01:15:38.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.725+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5618072330 con 0x7f56181008d0 2024-09-16T01:15:38.727 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:38 smithi078 ceph-mon[30937]: pgmap v148: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-09-16T01:15:38.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.727+0000 7f560f7fe640 1 -- 172.21.15.78:0/4236486717 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f560809c4b0 con 0x7f56181008d0 2024-09-16T01:15:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:15:38.729 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:15:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 msgr2=0x7f56040a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 0x7f56040a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f561806e210 tx=0x7f560000fae0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 msgr2=0x7f56180739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 0x7f56180739c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f560800b3f0 tx=0x7f560800b8b0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 shutdown_connections 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f56040a3430 0x7f56040a58f0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f561806e210 tx=0x7f560000fae0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56181068d0 0x7f561806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5618101270 0x7f561806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 --2- 172.21.15.78:0/4236486717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56181008d0 0x7f56180739c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.731+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 >> 172.21.15.78:0/4236486717 conn(0x7f56180fc610 msgr2=0x7f56180fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:38.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.732+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 shutdown_connections 2024-09-16T01:15:38.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:38.732+0000 7f561759e640 1 -- 172.21.15.78:0/4236486717 wait complete. 2024-09-16T01:15:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:38 smithi193 ceph-mon[35701]: pgmap v148: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-09-16T01:15:39.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:39 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4236486717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:39.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:39 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4236486717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:39.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:39 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4236486717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:15:40.529 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-16T01:15:40.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-16T01:15:40.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:40 smithi148 ceph-mon[35030]: pgmap v149: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-09-16T01:15:40.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:40 smithi078 ceph-mon[30937]: pgmap v149: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-09-16T01:15:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:40 smithi193 ceph-mon[35701]: pgmap v149: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-09-16T01:15:40.918 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:15:40.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 265.0 s, 1201.3 tps, lat 82.940 ms stddev 96.607 2024-09-16T01:15:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f87fff640 1 -- 172.21.15.78:0/438377013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8802f320 con 0x7f8f90101370 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 -- 172.21.15.78:0/438377013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f90101370 msgr2=0x7f8f9010d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 --2- 172.21.15.78:0/438377013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f90101370 0x7f8f9010d8a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8f88009cd0 tx=0x7f8f8802a910 comp rx=0 tx=0).stop 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 -- 172.21.15.78:0/438377013 shutdown_connections 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 --2- 172.21.15.78:0/438377013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f90101370 0x7f8f9010d8a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 --2- 172.21.15.78:0/438377013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 0x7f8f90100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 --2- 172.21.15.78:0/438377013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f901069e0 0x7f8f90106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 -- 172.21.15.78:0/438377013 >> 172.21.15.78:0/438377013 conn(0x7f8f900fc6f0 msgr2=0x7f8f900feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.271+0000 7f8f9741e640 1 -- 172.21.15.78:0/438377013 shutdown_connections 2024-09-16T01:15:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- 172.21.15.78:0/438377013 wait complete. 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 Processor -- start 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- start start 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 0x7f8f90198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 0x7f8f9019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90110e20 con 0x7f8f901069e0 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90110ca0 con 0x7f8f90101370 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f90110fa0 con 0x7f8f901009d0 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f959bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 0x7f8f9019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36222/0 (socket says 172.21.15.78:36222) 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 0x7f8f90198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 -- 172.21.15.78:0/3424609469 learned_addr learned my addr 172.21.15.78:0/3424609469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 -- 172.21.15.78:0/3424609469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 msgr2=0x7f8f90198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 0x7f8f90198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 -- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 msgr2=0x7f8f9019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 0x7f8f9019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 -- 172.21.15.78:0/3424609469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9019d100 con 0x7f8f90101370 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f959bb640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 0x7f8f9019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f949b9640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000c890 tx=0x7f8f8000cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f867fc640 1 -- 172.21.15.78:0/3424609469 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80004220 con 0x7f8f90101370 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f867fc640 1 -- 172.21.15.78:0/3424609469 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f800043c0 con 0x7f8f90101370 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f867fc640 1 -- 172.21.15.78:0/3424609469 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800049e0 con 0x7f8f90101370 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9019d3f0 con 0x7f8f90101370 2024-09-16T01:15:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.276+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f901a4c80 con 0x7f8f90101370 2024-09-16T01:15:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f867fc640 1 -- 172.21.15.78:0/3424609469 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f8f800079b0 con 0x7f8f90101370 2024-09-16T01:15:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 Processor -- start 2024-09-16T01:15:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- start start 2024-09-16T01:15:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f867fc640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 0x7f8f7c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 0x564d7ddfbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddfc0c0 0x564d7ddfe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564d7dd512f0 con 0x564d7ddf0f00 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564d7dcf9270 con 0x564d7ddfc0c0 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d7dd4d2b0 con 0x564d7ddf1820 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f867fc640 1 -- 172.21.15.78:0/3424609469 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8f8009a080 con 0x7f8f90101370 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45388/0 (socket says 172.21.15.78:45388) 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 -- 172.21.15.78:0/343115730 learned_addr learned my addr 172.21.15.78:0/343115730 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f959bb640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 0x7f8f7c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f959bb640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddfc0c0 0x564d7ddfe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f949b9640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 0x564d7ddfbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 -- 172.21.15.78:0/343115730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 msgr2=0x564d7ddfbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 0x564d7ddfbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 -- 172.21.15.78:0/343115730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddfc0c0 msgr2=0x564d7ddfe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddfc0c0 0x564d7ddfe4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 -- 172.21.15.78:0/343115730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d7dcf8340 con 0x564d7ddf0f00 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f949b9640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 0x564d7ddfbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f959bb640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 0x7f8f7c0a59a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005e30 tx=0x7f8f8802f3b0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f951ba640 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009c70 tx=0x7f8f8c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac79d02430cbbe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f7b7fe640 1 -- 172.21.15.78:0/343115730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c004230 con 0x564d7ddf0f00 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f7b7fe640 1 -- 172.21.15.78:0/343115730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c0043d0 con 0x564d7ddf0f00 2024-09-16T01:15:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f7b7fe640 1 -- 172.21.15.78:0/343115730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c019760 con 0x564d7ddf0f00 2024-09-16T01:15:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- 172.21.15.78:0/343115730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 msgr2=0x564d7ddf12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009c70 tx=0x7f8f8c011810 comp rx=0 tx=0).stop 2024-09-16T01:15:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- 172.21.15.78:0/343115730 shutdown_connections 2024-09-16T01:15:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddfc0c0 0x564d7ddfe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf1820 0x564d7ddfbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/343115730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf0f00 0x564d7ddf12e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- 172.21.15.78:0/343115730 >> 172.21.15.78:0/343115730 conn(0x564d7dded470 msgr2=0x564d7dded850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- 172.21.15.78:0/343115730 shutdown_connections 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.281+0000 7f8f9a28c740 1 -- 172.21.15.78:0/343115730 wait complete. 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 -- start start 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 Processor -- start 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7ddf5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7ddf5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564d7dde8c10 con 0x564d7ddfc0c0 2024-09-16T01:15:41.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564d7dcda290 con 0x564d7ddf1820 2024-09-16T01:15:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564d7dde8d90 con 0x564d7ddf0f00 2024-09-16T01:15:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 10 client.0 _ll_register_callbacks cb 0x564d7dd0b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36240/0 (socket says 172.21.15.78:36240) 2024-09-16T01:15:41.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 -- 172.21.15.78:0/2540907153 learned_addr learned my addr 172.21.15.78:0/2540907153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f959bb640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7ddf5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f951ba640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7ddf5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 -- 172.21.15.78:0/2540907153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 msgr2=0x564d7ddf5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7ddf5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 -- 172.21.15.78:0/2540907153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 msgr2=0x564d7ddf5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7ddf5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 -- 172.21.15.78:0/2540907153 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d7da64210 con 0x564d7ddf1820 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f959bb640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7ddf5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f949b9640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f800054e0 tx=0x7f8f8009bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18f900aa2351cc0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8001d040 con 0x564d7ddf1820 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8f800a4710 con 0x564d7ddf1820 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.286+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f8c0043d0 con 0x564d7ddf1820 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f800cd830 con 0x564d7ddf1820 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 10 client.25427 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d7db30620 con 0x564d7ddf1820 2024-09-16T01:15:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c0043d0 con 0x564d7ddf1820 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8f800bab70 con 0x564d7ddf1820 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 10 client.25427 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 10 client.25427 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 20 client.25427 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de10800 0x564d7de12ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 4 client.25427 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:41.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.291+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d7dd5ea20 con 0x564d7de10800 2024-09-16T01:15:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.296+0000 7f8f951ba640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de10800 0x564d7de12ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.310+0000 7f8f951ba640 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de10800 0x564d7de12ca0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f70fcc7d80392456 server_cookie=40b7e56fcf91a9b6 in_seq=0 out_seq=0 2024-09-16T01:15:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.310+0000 7f8f78ff9640 10 client.25427 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.315+0000 7f8f78ff9640 1 -- 172.21.15.78:0/2540907153 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564d7dd5ea20 con 0x564d7de10800 2024-09-16T01:15:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.315+0000 7f8f78ff9640 10 client.25427 handle_command_reply: tid=0 2024-09-16T01:15:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.315+0000 7f8f9a28c740 1 client.25427 shutdown 2024-09-16T01:15:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.319+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 msgr2=0x564d7ddf56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.319+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f800054e0 tx=0x7f8f8009bc50 comp rx=0 tx=0).stop 2024-09-16T01:15:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 shutdown_connections 2024-09-16T01:15:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de10800 0x564d7de12ca0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7ddf5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7ddf56c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/2540907153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7ddf5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 >> 172.21.15.78:0/2540907153 conn(0x564d7dded470 msgr2=0x564d7dde81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.323+0000 7f8f78ff9640 0 client.25427 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.326+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 shutdown_connections 2024-09-16T01:15:41.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.326+0000 7f8f9a28c740 1 -- 172.21.15.78:0/2540907153 wait complete. 2024-09-16T01:15:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.329+0000 7f8f9a28c740 20 client.25427 trim_cache size 0 max 16384 2024-09-16T01:15:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.371+0000 7f8f9a28c740 1 Processor -- start 2024-09-16T01:15:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.372+0000 7f8f9a28c740 1 -- start start 2024-09-16T01:15:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf1820 0x564d7de185b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36242/0 (socket says 172.21.15.78:36242) 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f949b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf1820 0x564d7de185b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f951ba640 1 -- 172.21.15.78:0/1596468600 learned_addr learned my addr 172.21.15.78:0/1596468600 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.373+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddfc0c0 0x564d7de18af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.374+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901af6f0 con 0x564d7ddf1820 2024-09-16T01:15:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.374+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901acfc0 con 0x564d7ddf0f00 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.374+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901aae10 con 0x564d7ddfc0c0 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f959bb640 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddfc0c0 0x564d7de18af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f951ba640 1 -- 172.21.15.78:0/1596468600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddfc0c0 msgr2=0x564d7de18af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f951ba640 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddfc0c0 0x564d7de18af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f951ba640 1 -- 172.21.15.78:0/1596468600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf1820 msgr2=0x564d7de185b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f951ba640 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf1820 0x564d7de185b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.375+0000 7f8f951ba640 1 -- 172.21.15.78:0/1596468600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d7de13320 con 0x564d7ddf0f00 2024-09-16T01:15:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.376+0000 7f8f951ba640 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009c70 tx=0x7f8f8c003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac537ee08a24817 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.378+0000 7f8f7affd640 1 -- 172.21.15.78:0/1596468600 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c02a070 con 0x564d7ddf0f00 2024-09-16T01:15:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.378+0000 7f8f7affd640 1 -- 172.21.15.78:0/1596468600 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c020040 con 0x564d7ddf0f00 2024-09-16T01:15:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.378+0000 7f8f7affd640 1 -- 172.21.15.78:0/1596468600 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c01f040 con 0x564d7ddf0f00 2024-09-16T01:15:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.380+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 msgr2=0x564d7ddf9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.380+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009c70 tx=0x7f8f8c003e60 comp rx=0 tx=0).stop 2024-09-16T01:15:41.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.383+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 shutdown_connections 2024-09-16T01:15:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.383+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddfc0c0 0x564d7de18af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.383+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddf1820 0x564d7de185b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.383+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/1596468600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf0f00 0x564d7ddf9100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.383+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 >> 172.21.15.78:0/1596468600 conn(0x564d7dded470 msgr2=0x564d7ddf89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.385+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 shutdown_connections 2024-09-16T01:15:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.388+0000 7f8f9a28c740 1 -- 172.21.15.78:0/1596468600 wait complete. 2024-09-16T01:15:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.388+0000 7f8f9a28c740 1 -- start start 2024-09-16T01:15:41.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.389+0000 7f8f9a28c740 1 Processor -- start 2024-09-16T01:15:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7de07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7de084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901ab480 con 0x564d7ddfc0c0 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901ac8f0 con 0x564d7ddf1820 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f901abb50 con 0x564d7ddf0f00 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f951ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:37840/0 (socket says 172.21.15.78:37840) 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f951ba640 1 -- 172.21.15.78:0/213088387 learned_addr learned my addr 172.21.15.78:0/213088387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f9a28c740 10 client.0 _ll_register_callbacks cb 0x7f8f800046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:41.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.390+0000 7f8f959bb640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7de084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f949b9640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7de07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f951ba640 1 -- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 msgr2=0x564d7de07f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f951ba640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7de07f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f951ba640 1 -- 172.21.15.78:0/213088387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 msgr2=0x564d7de084d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f951ba640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7de084d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.391+0000 7f8f951ba640 1 -- 172.21.15.78:0/213088387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d7ddf5a30 con 0x564d7ddf0f00 2024-09-16T01:15:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.392+0000 7f8f951ba640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011850 tx=0x7f8f8c009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb538c47e70bd70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.392+0000 7f8f78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:15:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.393+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c02a040 con 0x564d7ddf0f00 2024-09-16T01:15:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.393+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f8f8c0035f0 con 0x564d7ddf0f00 2024-09-16T01:15:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.394+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f8c020040 con 0x564d7ddf0f00 2024-09-16T01:15:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.395+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f8c0037f0 con 0x564d7ddf0f00 2024-09-16T01:15:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.395+0000 7f8f9a28c740 10 client.25462 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.395+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d7ddee100 con 0x564d7ddf0f00 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.395+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c020040 con 0x564d7ddf0f00 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.395+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8f8c018070 con 0x564d7ddf0f00 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 10 client.25462 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 10 client.25462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 20 client.25462 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de7b4a0 0x564d7de7d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 4 client.25462 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d7dd58d70 con 0x564d7de7b4a0 2024-09-16T01:15:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.396+0000 7f8f949b9640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de7b4a0 0x564d7de7d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.404+0000 7f8f949b9640 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de7b4a0 0x564d7de7d890 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a93f21843a68f70 server_cookie=2f130998596a832b in_seq=0 out_seq=0 2024-09-16T01:15:41.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.404+0000 7f8f78ff9640 10 client.25462 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.422+0000 7f8f78ff9640 1 -- 172.21.15.78:0/213088387 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d7dd58d70 con 0x564d7de7b4a0 2024-09-16T01:15:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.422+0000 7f8f78ff9640 10 client.25462 handle_command_reply: tid=0 2024-09-16T01:15:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.422+0000 7f8f9a28c740 1 client.25462 shutdown 2024-09-16T01:15:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.422+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 msgr2=0x564d7de0cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.422+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011850 tx=0x7f8f8c009370 comp rx=0 tx=0).stop 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 shutdown_connections 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564d7de7b4a0 0x564d7de7d890 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564d7ddfc0c0 0x564d7de084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d7ddf1820 0x564d7de07f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 --2- 172.21.15.78:0/213088387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d7ddf0f00 0x564d7de0cc70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 >> 172.21.15.78:0/213088387 conn(0x564d7dded470 msgr2=0x564d7ddf5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f78ff9640 0 client.25462 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.424+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 shutdown_connections 2024-09-16T01:15:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.425+0000 7f8f9a28c740 1 -- 172.21.15.78:0/213088387 wait complete. 2024-09-16T01:15:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.425+0000 7f8f9a28c740 20 client.25462 trim_cache size 0 max 16384 2024-09-16T01:15:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:15:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:15:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:15:41.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:15:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.429+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 msgr2=0x7f8f7c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.429+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 0x7f8f7c0a59a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005e30 tx=0x7f8f8802f3b0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.429+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 msgr2=0x7f8f90198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.429+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000c890 tx=0x7f8f8000cd50 comp rx=0 tx=0).stop 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.429+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 shutdown_connections 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8f7c0a34e0 0x7f8f7c0a59a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f901069e0 0x7f8f9019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f90101370 0x7f8f90198650 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 --2- 172.21.15.78:0/3424609469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f901009d0 0x7f8f90198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 >> 172.21.15.78:0/3424609469 conn(0x7f8f900fc6f0 msgr2=0x7f8f90104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 shutdown_connections 2024-09-16T01:15:41.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:41.430+0000 7f8f9741e640 1 -- 172.21.15.78:0/3424609469 wait complete. 2024-09-16T01:15:41.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.556+0000 7f69392e1640 1 -- 172.21.15.78:0/1324682830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109e00 msgr2=0x7f69340742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.556+0000 7f69392e1640 1 --2- 172.21.15.78:0/1324682830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109e00 0x7f69340742d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f692c00afd0 tx=0x7f692c02b400 comp rx=0 tx=0).stop 2024-09-16T01:15:41.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.558+0000 7f69392e1640 1 -- 172.21.15.78:0/1324682830 shutdown_connections 2024-09-16T01:15:41.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.558+0000 7f69392e1640 1 --2- 172.21.15.78:0/1324682830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934074990 0x7f693407b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.558+0000 7f69392e1640 1 --2- 172.21.15.78:0/1324682830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109e00 0x7f69340742d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.558+0000 7f69392e1640 1 --2- 172.21.15.78:0/1324682830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6934109450 0x7f6934109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.558+0000 7f69392e1640 1 -- 172.21.15.78:0/1324682830 >> 172.21.15.78:0/1324682830 conn(0x7f693406e140 msgr2=0x7f693406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.559+0000 7f69392e1640 1 -- 172.21.15.78:0/1324682830 shutdown_connections 2024-09-16T01:15:41.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.561+0000 7f69392e1640 1 -- 172.21.15.78:0/1324682830 wait complete. 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 Processor -- start 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 -- start start 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6934074990 0x7f693407b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109450 0x7f6934076310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6934084da0 con 0x7f6934109450 2024-09-16T01:15:41.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6934084c20 con 0x7f6934109e00 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.563+0000 7f69392e1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6934084f20 con 0x7f6934074990 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.564+0000 7f69337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.564+0000 7f69337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36264/0 (socket says 172.21.15.78:36264) 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.564+0000 7f69337fe640 1 -- 172.21.15.78:0/2551495103 learned_addr learned my addr 172.21.15.78:0/2551495103 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69327fc640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109450 0x7f6934076310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69337fe640 1 -- 172.21.15.78:0/2551495103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6934074990 msgr2=0x7f693407b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69337fe640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6934074990 0x7f693407b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69337fe640 1 -- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109450 msgr2=0x7f6934076310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69337fe640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109450 0x7f6934076310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.565+0000 7f69337fe640 1 -- 172.21.15.78:0/2551495103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6934077130 con 0x7f6934109e00 2024-09-16T01:15:41.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.566+0000 7f69337fe640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f692800d750 tx=0x7f692800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.567+0000 7f691bfff640 1 -- 172.21.15.78:0/2551495103 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928017070 con 0x7f6934109e00 2024-09-16T01:15:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.567+0000 7f691bfff640 1 -- 172.21.15.78:0/2551495103 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6928005500 con 0x7f6934109e00 2024-09-16T01:15:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.568+0000 7f691bfff640 1 -- 172.21.15.78:0/2551495103 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928004d60 con 0x7f6934109e00 2024-09-16T01:15:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.568+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69341373b0 con 0x7f6934109e00 2024-09-16T01:15:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.568+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6934137870 con 0x7f6934109e00 2024-09-16T01:15:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.571+0000 7f691bfff640 1 -- 172.21.15.78:0/2551495103 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6928004f00 con 0x7f6934109e00 2024-09-16T01:15:41.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.575+0000 7f691bfff640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 0x7f691c0a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.575+0000 7f693b6ba740 1 Processor -- start 2024-09-16T01:15:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.575+0000 7f691bfff640 1 -- 172.21.15.78:0/2551495103 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69280cfa70 con 0x7f6934109e00 2024-09-16T01:15:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.575+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 0x7f691c0a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 -- start start 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02ab850 0x562de02abc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562de0210160 con 0x562de02ab850 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562de01b7720 con 0x562de02ac170 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f693b6ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562de020c120 con 0x562de02b6a30 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f69337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f69337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:37864/0 (socket says 172.21.15.78:37864) 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f69337fe640 1 -- 172.21.15.78:0/2812057378 learned_addr learned my addr 172.21.15.78:0/2812057378 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f69327fc640 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.576+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02ab850 0x562de02abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 -- 172.21.15.78:0/2812057378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 msgr2=0x562de02b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 -- 172.21.15.78:0/2812057378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02ab850 msgr2=0x562de02abc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02ab850 0x562de02abc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 -- 172.21.15.78:0/2812057378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ddff81ba0 con 0x562de02b6a30 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 0x7f691c0a5a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6924009ad0 tx=0x7f6924009590 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69337fe640 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6928098390 tx=0x7f69280057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=335a594813ce5ec9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.577+0000 7f69197fa640 1 -- 172.21.15.78:0/2812057378 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928020040 con 0x562de02b6a30 2024-09-16T01:15:41.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.578+0000 7f69197fa640 1 -- 172.21.15.78:0/2812057378 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69280a4630 con 0x562de02b6a30 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.578+0000 7f69197fa640 1 -- 172.21.15.78:0/2812057378 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928098030 con 0x562de02b6a30 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.578+0000 7f693b6ba740 1 -- 172.21.15.78:0/2812057378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 msgr2=0x562de02b8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.578+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6928098390 tx=0x7f69280057f0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 -- 172.21.15.78:0/2812057378 shutdown_connections 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b6a30 0x562de02b8e20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2812057378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02ab850 0x562de02abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 -- 172.21.15.78:0/2812057378 >> 172.21.15.78:0/2812057378 conn(0x562de02a7de0 msgr2=0x562de02a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 -- 172.21.15.78:0/2812057378 shutdown_connections 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 -- 172.21.15.78:0/2812057378 wait complete. 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 -- start start 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.579+0000 7f693b6ba740 1 Processor -- start 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 0x562de02b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562de02b96a0 con 0x562de02b6a30 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562de0198900 con 0x562de02ac170 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562de02b9820 con 0x562de02ab850 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f693b6ba740 10 client.0 _ll_register_callbacks cb 0x562de0196780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f69327fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f69327fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36284/0 (socket says 172.21.15.78:36284) 2024-09-16T01:15:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f69327fc640 1 -- 172.21.15.78:0/2993368246 learned_addr learned my addr 172.21.15.78:0/2993368246 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.580+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 0x562de02b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 -- 172.21.15.78:0/2993368246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 msgr2=0x562de02b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 0x562de02b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 -- 172.21.15.78:0/2993368246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 msgr2=0x562de02b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 -- 172.21.15.78:0/2993368246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562de01d8c50 con 0x562de02ac170 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 0x562de02b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.581+0000 7f69327fc640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f692c005e30 tx=0x7f692c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c2fe82c29a94786 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f6916ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c043070 con 0x562de02ac170 2024-09-16T01:15:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69280a4630 con 0x562de02ac170 2024-09-16T01:15:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f692c032d80 con 0x562de02ac170 2024-09-16T01:15:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f692c04f050 con 0x562de02ac170 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f693b6ba740 10 client.25442 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.582+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562de02c5340 con 0x562de02ac170 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69280a4630 con 0x562de02ac170 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f692c004040 con 0x562de02ac170 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 10 client.25442 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 10 client.25442 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 20 client.25442 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de02c6810 0x562de02c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 4 client.25442 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.583+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562de021d890 con 0x562de02c6810 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.584+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de02c6810 0x562de02c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.584+0000 7f6932ffd640 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de02c6810 0x562de02c8c70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb71814a43fb4c7f server_cookie=1e299197fd89175 in_seq=0 out_seq=0 2024-09-16T01:15:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.584+0000 7f6916ff5640 10 client.25442 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f6916ff5640 1 -- 172.21.15.78:0/2993368246 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562de021d890 con 0x562de02c6810 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f6916ff5640 10 client.25442 handle_command_reply: tid=0 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 client.25442 shutdown 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 msgr2=0x562de02b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f692c005e30 tx=0x7f692c004a40 comp rx=0 tx=0).stop 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 shutdown_connections 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de02c6810 0x562de02c8c70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ac170 0x562de02b3db0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 --2- 172.21.15.78:0/2993368246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02ab850 0x562de02b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 >> 172.21.15.78:0/2993368246 conn(0x562de02a7de0 msgr2=0x562de02adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f6916ff5640 0 client.25442 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 shutdown_connections 2024-09-16T01:15:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.585+0000 7f693b6ba740 1 -- 172.21.15.78:0/2993368246 wait complete. 2024-09-16T01:15:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.586+0000 7f693b6ba740 20 client.25442 trim_cache size 0 max 16384 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.610+0000 7f693b6ba740 1 Processor -- start 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.610+0000 7f693b6ba740 1 -- start start 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d9f90 con 0x562de02b6a30 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d7860 con 0x562de02ab850 2024-09-16T01:15:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f693b6ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d56b0 con 0x562de02b3830 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f6932ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45450/0 (socket says 172.21.15.78:45450) 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69337fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:37884/0 (socket says 172.21.15.78:37884) 2024-09-16T01:15:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 -- 172.21.15.78:0/4176405150 learned_addr learned my addr 172.21.15.78:0/4176405150 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 -- 172.21.15.78:0/4176405150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 msgr2=0x562de02b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 -- 172.21.15.78:0/4176405150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 msgr2=0x562de02ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 -- 172.21.15.78:0/4176405150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562de02a7c10 con 0x562de02b6a30 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f6932ffd640 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.611+0000 7f69327fc640 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f692c004560 tx=0x7f692c007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6b67edcd3b4403 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.612+0000 7f6918ff9640 1 -- 172.21.15.78:0/4176405150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c043050 con 0x562de02b6a30 2024-09-16T01:15:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.612+0000 7f6918ff9640 1 -- 172.21.15.78:0/4176405150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f692c038040 con 0x562de02b6a30 2024-09-16T01:15:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.612+0000 7f6918ff9640 1 -- 172.21.15.78:0/4176405150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692c037040 con 0x562de02b6a30 2024-09-16T01:15:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 -- 172.21.15.78:0/4176405150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 msgr2=0x562de02cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f692c004560 tx=0x7f692c007340 comp rx=0 tx=0).stop 2024-09-16T01:15:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 -- 172.21.15.78:0/4176405150 shutdown_connections 2024-09-16T01:15:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02cea80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 --2- 172.21.15.78:0/4176405150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 -- 172.21.15.78:0/4176405150 >> 172.21.15.78:0/4176405150 conn(0x562de02a7de0 msgr2=0x562de02b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 -- 172.21.15.78:0/4176405150 shutdown_connections 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.613+0000 7f693b6ba740 1 -- 172.21.15.78:0/4176405150 wait complete. 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 -- start start 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 Processor -- start 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d5d20 con 0x562de02b6a30 2024-09-16T01:15:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d7190 con 0x562de02ab850 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69341d63f0 con 0x562de02b3830 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f693b6ba740 10 client.0 _ll_register_callbacks cb 0x7f6924005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f69337fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.614+0000 7f6932ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36296/0 (socket says 172.21.15.78:36296) 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 -- 172.21.15.78:0/1393000160 learned_addr learned my addr 172.21.15.78:0/1393000160 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f69327fc640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 -- 172.21.15.78:0/1393000160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 msgr2=0x562de02bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02bf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 -- 172.21.15.78:0/1393000160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 msgr2=0x562de02b9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 -- 172.21.15.78:0/1393000160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562de02ba2b0 con 0x562de02ab850 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f69327fc640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6916ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6932ffd640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6924005d60 tx=0x7f6924005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14a229e4f251ea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924027830 con 0x562de02ab850 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.615+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f69240279d0 con 0x562de02ab850 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f692c038040 con 0x562de02ab850 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6924027bd0 con 0x562de02ab850 2024-09-16T01:15:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 10 client.25448 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562de0332030 con 0x562de02ab850 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f692c038040 con 0x562de02ab850 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f692403b540 con 0x562de02ab850 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 10 client.25448 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 10 client.25448 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 20 client.25448 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de03327c0 0x562de0334c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 4 client.25448 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x562de02a3410 con 0x562de03327c0 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.616+0000 7f69327fc640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de03327c0 0x562de0334c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.617+0000 7f6916ff5640 10 client.25448 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.617+0000 7f69327fc640 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de03327c0 0x562de0334c00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceb8a690be9f7bcd server_cookie=677e71f91ee10f61 in_seq=0 out_seq=0 2024-09-16T01:15:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.970+0000 7f6916ff5640 1 -- 172.21.15.78:0/1393000160 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562de02a3410 con 0x562de03327c0 2024-09-16T01:15:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.970+0000 7f6916ff5640 10 client.25448 handle_command_reply: tid=0 2024-09-16T01:15:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.970+0000 7f693b6ba740 1 client.25448 shutdown 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 msgr2=0x562de02becb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6924005d60 tx=0x7f6924005f90 comp rx=0 tx=0).stop 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 shutdown_connections 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562de03327c0 0x562de0334c00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562de02b6a30 0x562de02b9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562de02b3830 0x562de02bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f6916ff5640 0 client.25448 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 --2- 172.21.15.78:0/1393000160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562de02ab850 0x562de02becb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 >> 172.21.15.78:0/1393000160 conn(0x562de02a7de0 msgr2=0x562de02b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 shutdown_connections 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 1 -- 172.21.15.78:0/1393000160 wait complete. 2024-09-16T01:15:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f693b6ba740 20 client.25448 trim_cache size 0 max 16384 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 18, 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "5be7c540": { 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 18, 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:15:41.974 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 145.3, 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:41.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 msgr2=0x7f691c0a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 0x7f691c0a5a00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6924009ad0 tx=0x7f6924009590 comp rx=0 tx=0).stop 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 msgr2=0x7f6934076870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f692800d750 tx=0x7f692800dc10 comp rx=0 tx=0).stop 2024-09-16T01:15:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 shutdown_connections 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f691c0a3540 0x7f691c0a5a00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.971+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6934109e00 0x7f6934076870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.972+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6934109450 0x7f6934076310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.972+0000 7f69392e1640 1 --2- 172.21.15.78:0/2551495103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6934074990 0x7f693407b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.972+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 >> 172.21.15.78:0/2551495103 conn(0x7f693406e140 msgr2=0x7f6934071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.972+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 shutdown_connections 2024-09-16T01:15:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:15:41.972+0000 7f69392e1640 1 -- 172.21.15.78:0/2551495103 wait complete. 2024-09-16T01:15:42.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:42 smithi148 ceph-mon[35030]: pgmap v150: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-09-16T01:15:42.673 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:42 smithi078 ceph-mon[30937]: pgmap v150: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-09-16T01:15:42.700 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5be7c540', quiesce took 5.4 seconds. Will release after: 19.950000000000003 2024-09-16T01:15:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:42 smithi193 ceph-mon[35701]: pgmap v150: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-09-16T01:15:44.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:44 smithi148 ceph-mon[35030]: pgmap v151: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 15 MiB/s wr, 950 op/s 2024-09-16T01:15:44.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:44 smithi078 ceph-mon[30937]: pgmap v151: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 15 MiB/s wr, 950 op/s 2024-09-16T01:15:44.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:44 smithi193 ceph-mon[35701]: pgmap v151: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 15 MiB/s wr, 950 op/s 2024-09-16T01:15:45.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 270.0 s, 159.1 tps, lat 96.814 ms stddev 88.109 2024-09-16T01:15:46.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:46 smithi078 ceph-mon[30937]: pgmap v152: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:46.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:46 smithi193 ceph-mon[35701]: pgmap v152: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:46.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:46 smithi148 ceph-mon[35030]: pgmap v152: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-09-16T01:15:48.780 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:48 smithi078 ceph-mon[30937]: pgmap v153: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 501 op/s 2024-09-16T01:15:48.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:48 smithi193 ceph-mon[35701]: pgmap v153: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 501 op/s 2024-09-16T01:15:48.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:48 smithi148 ceph-mon[35030]: pgmap v153: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 501 op/s 2024-09-16T01:15:50.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:50 smithi078 ceph-mon[30937]: pgmap v154: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 538 op/s 2024-09-16T01:15:50.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:50 smithi193 ceph-mon[35701]: pgmap v154: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 538 op/s 2024-09-16T01:15:50.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:50 smithi148 ceph-mon[35030]: pgmap v154: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 538 op/s 2024-09-16T01:15:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 275.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:15:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:52 smithi078 ceph-mon[30937]: pgmap v155: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.3 MiB/s wr, 301 op/s 2024-09-16T01:15:52.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:52 smithi193 ceph-mon[35701]: pgmap v155: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.3 MiB/s wr, 301 op/s 2024-09-16T01:15:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:52 smithi148 ceph-mon[35030]: pgmap v155: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.3 MiB/s wr, 301 op/s 2024-09-16T01:15:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.602+0000 7fd3af9c4640 1 -- 172.21.15.78:0/131292373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 msgr2=0x7fd3a8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.602+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/131292373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a8110250 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009ca0 tx=0x7fd3a002a8d0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.603+0000 7fd3af9c4640 1 -- 172.21.15.78:0/131292373 shutdown_connections 2024-09-16T01:15:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.603+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/131292373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 0x7fd3a811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.603+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/131292373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 0x7fd3a8109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.603+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/131292373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a8110250 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.603+0000 7fd3af9c4640 1 -- 172.21.15.78:0/131292373 >> 172.21.15.78:0/131292373 conn(0x7fd3a806ce00 msgr2=0x7fd3a806d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.607+0000 7fd3af9c4640 1 -- 172.21.15.78:0/131292373 shutdown_connections 2024-09-16T01:15:54.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.607+0000 7fd3af9c4640 1 -- 172.21.15.78:0/131292373 wait complete. 2024-09-16T01:15:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.607+0000 7fd3af9c4640 1 Processor -- start 2024-09-16T01:15:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.612+0000 7fd3af9c4640 1 -- start start 2024-09-16T01:15:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 0x7fd3a81a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 0x7fd3a81aba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8120d50 con 0x7fd3a81091f0 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8120bd0 con 0x7fd3a811b140 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.613+0000 7fd3af9c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8120ed0 con 0x7fd3a81095d0 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.615+0000 7fd3ad760640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.615+0000 7fd3ad760640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42058/0 (socket says 172.21.15.78:42058) 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.615+0000 7fd3ad760640 1 -- 172.21.15.78:0/529058932 learned_addr learned my addr 172.21.15.78:0/529058932 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.615+0000 7fd3adf61640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 0x7fd3a81aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.615+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 0x7fd3a81a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.619+0000 7fd3ad760640 1 -- 172.21.15.78:0/529058932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 msgr2=0x7fd3a81a76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.619+0000 7fd3ad760640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 0x7fd3a81a76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.619+0000 7fd3ad760640 1 -- 172.21.15.78:0/529058932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 msgr2=0x7fd3a81aba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.619+0000 7fd3ad760640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 0x7fd3a81aba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.619+0000 7fd3ad760640 1 -- 172.21.15.78:0/529058932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a81ac150 con 0x7fd3a81091f0 2024-09-16T01:15:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.623+0000 7fd3ad760640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd3a002aec0 tx=0x7fd3a0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.626+0000 7fd39e7fc640 1 -- 172.21.15.78:0/529058932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0031660 con 0x7fd3a81091f0 2024-09-16T01:15:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.626+0000 7fd39e7fc640 1 -- 172.21.15.78:0/529058932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a0031800 con 0x7fd3a81091f0 2024-09-16T01:15:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.626+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a81ac3e0 con 0x7fd3a81091f0 2024-09-16T01:15:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.626+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a8110920 con 0x7fd3a81091f0 2024-09-16T01:15:54.629 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.628+0000 7fd39e7fc640 1 -- 172.21.15.78:0/529058932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002fab0 con 0x7fd3a81091f0 2024-09-16T01:15:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.630+0000 7fd3b2838740 1 Processor -- start 2024-09-16T01:15:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 -- start start 2024-09-16T01:15:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd29ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd34770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fd370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b97fc8e2f0 con 0x55b97fd29af0 2024-09-16T01:15:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b97fc36270 con 0x55b97fd2a410 2024-09-16T01:15:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.633+0000 7fd3b2838740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b97fc8a2b0 con 0x55b97fd34cb0 2024-09-16T01:15:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd39e7fc640 1 -- 172.21.15.78:0/529058932 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fd3a002fc50 con 0x7fd3a81091f0 2024-09-16T01:15:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd3adf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd29ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd3adf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd29ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42062/0 (socket says 172.21.15.78:42062) 2024-09-16T01:15:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd3adf61640 1 -- 172.21.15.78:0/946455924 learned_addr learned my addr 172.21.15.78:0/946455924 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fd370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.634+0000 7fd3ad760640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd34770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd39e7fc640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 0x7fd3940a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd39e7fc640 1 -- 172.21.15.78:0/529058932 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd3a00f1340 con 0x7fd3a81091f0 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd3acf5f640 1 -- 172.21.15.78:0/946455924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 msgr2=0x55b97fd34770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd34770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd3acf5f640 1 -- 172.21.15.78:0/946455924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 msgr2=0x55b97fd29ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd29ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.641+0000 7fd3acf5f640 1 -- 172.21.15.78:0/946455924 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b97fc35340 con 0x55b97fd34cb0 2024-09-16T01:15:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.642+0000 7fd3ad760640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd34770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:15:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.642+0000 7fd3adf61640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 0x7fd3940a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.643+0000 7fd3adf61640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 0x7fd3940a5b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40045e0 tx=0x7fd3a40046c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.645+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fd370a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd39800b730 tx=0x7fd39800bbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebc070279f0865e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.647+0000 7fd3937fe640 1 -- 172.21.15.78:0/946455924 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39800be40 con 0x55b97fd34cb0 2024-09-16T01:15:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.647+0000 7fd3937fe640 1 -- 172.21.15.78:0/946455924 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398019070 con 0x55b97fd34cb0 2024-09-16T01:15:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.648+0000 7fd3b2838740 1 -- 172.21.15.78:0/946455924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 msgr2=0x55b97fd370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.648+0000 7fd3b2838740 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fd370a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd39800b730 tx=0x7fd39800bbf0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.650+0000 7fd3b2838740 1 -- 172.21.15.78:0/946455924 shutdown_connections 2024-09-16T01:15:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.650+0000 7fd3b2838740 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fd370a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.650+0000 7fd3b2838740 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd34770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.650+0000 7fd3b2838740 1 --2- 172.21.15.78:0/946455924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd29ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.650+0000 7fd3b2838740 1 -- 172.21.15.78:0/946455924 >> 172.21.15.78:0/946455924 conn(0x55b97fd26060 msgr2=0x55b97fd26440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.652+0000 7fd3b2838740 1 -- 172.21.15.78:0/946455924 shutdown_connections 2024-09-16T01:15:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.656+0000 7fd3b2838740 1 -- 172.21.15.78:0/946455924 wait complete. 2024-09-16T01:15:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.657+0000 7fd3b2838740 1 -- start start 2024-09-16T01:15:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.657+0000 7fd3b2838740 1 Processor -- start 2024-09-16T01:15:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd29af0 0x55b97fd31ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd2a410 0x55b97fd32020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b97fd378d0 con 0x55b97fd34cb0 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b97fc17290 con 0x55b97fd29af0 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b97fd37a50 con 0x55b97fd2a410 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.659+0000 7fd3b2838740 10 client.0 _ll_register_callbacks cb 0x55b97fc48140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.660+0000 7fd3adf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.660+0000 7fd3adf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42066/0 (socket says 172.21.15.78:42066) 2024-09-16T01:15:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.660+0000 7fd3adf61640 1 -- 172.21.15.78:0/232316139 learned_addr learned my addr 172.21.15.78:0/232316139 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.661+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd2a410 0x55b97fd32020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 -- 172.21.15.78:0/232316139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd2a410 msgr2=0x55b97fd32020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd2a410 0x55b97fd32020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 -- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd29af0 msgr2=0x55b97fd31ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:15:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd29af0 0x55b97fd31ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 -- 172.21.15.78:0/232316139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b97f9a1210 con 0x55b97fd34cb0 2024-09-16T01:15:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3adf61640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40306a0 tx=0x7fd3a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fac4da1338a3274 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:15:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4021500 con 0x55b97fd34cb0 2024-09-16T01:15:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.662+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd398019070 con 0x55b97fd34cb0 2024-09-16T01:15:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.663+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd3a40216a0 con 0x55b97fd34cb0 2024-09-16T01:15:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.663+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd3a4044050 con 0x55b97fd34cb0 2024-09-16T01:15:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.665+0000 7fd3b2838740 10 client.15999 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:54.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.665+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b97fa6d620 con 0x55b97fd34cb0 2024-09-16T01:15:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.669+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398019070 con 0x55b97fd34cb0 2024-09-16T01:15:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd3a402d6a0 con 0x55b97fd34cb0 2024-09-16T01:15:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 10 client.15999 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 20 client.15999 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fd44c40 0x55b97fd470b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 4 client.15999 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:15:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.670+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b97fc9ba20 con 0x55b97fd44c40 2024-09-16T01:15:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.671+0000 7fd3ad760640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fd44c40 0x55b97fd470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.673+0000 7fd3ad760640 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fd44c40 0x55b97fd470b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e6e98b0641ce293 server_cookie=87bf7030b3c1964e in_seq=0 out_seq=0 2024-09-16T01:15:54.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.673+0000 7fd390ff9640 10 client.15999 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.676+0000 7fd390ff9640 1 -- 172.21.15.78:0/232316139 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b97fc9ba20 con 0x55b97fd44c40 2024-09-16T01:15:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.676+0000 7fd390ff9640 10 client.15999 handle_command_reply: tid=0 2024-09-16T01:15:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.676+0000 7fd3b2838740 1 client.15999 shutdown 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.680+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 msgr2=0x55b97fd32560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.680+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40306a0 tx=0x7fd3a4004730 comp rx=0 tx=0).stop 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 shutdown_connections 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fd44c40 0x55b97fd470b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd34cb0 0x55b97fd32560 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd2a410 0x55b97fd32020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 --2- 172.21.15.78:0/232316139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd29af0 0x55b97fd31ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.681+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 >> 172.21.15.78:0/232316139 conn(0x55b97fd26060 msgr2=0x55b97fd2bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.682+0000 7fd390ff9640 0 client.15999 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.684+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 shutdown_connections 2024-09-16T01:15:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.686+0000 7fd3b2838740 1 -- 172.21.15.78:0/232316139 wait complete. 2024-09-16T01:15:54.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.688+0000 7fd3b2838740 20 client.15999 trim_cache size 0 max 16384 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.724+0000 7fd3b2838740 1 Processor -- start 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3b2838740 1 -- start start 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd29af0 0x55b97fd32390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd34cb0 0x55b97fd4cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3acf5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.725+0000 7fd3acf5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42068/0 (socket says 172.21.15.78:42068) 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.726+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3181983680 learned_addr learned my addr 172.21.15.78:0/3181983680 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.726+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c012780 con 0x55b97fd2a410 2024-09-16T01:15:54.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.726+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c010050 con 0x55b97fd34cb0 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.726+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c00df50 con 0x55b97fd29af0 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3181983680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd29af0 msgr2=0x55b97fd32390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd29af0 0x55b97fd32390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3181983680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd34cb0 msgr2=0x55b97fd4cf00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd34cb0 0x55b97fd4cf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3181983680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b97fd2aa30 con 0x55b97fd2a410 2024-09-16T01:15:54.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.727+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55b97fd2c690 tx=0x7fd398014b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58249bc6b64d3d89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.729+0000 7fd392ffd640 1 -- 172.21.15.78:0/3181983680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398014630 con 0x55b97fd2a410 2024-09-16T01:15:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.729+0000 7fd392ffd640 1 -- 172.21.15.78:0/3181983680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398019040 con 0x55b97fd2a410 2024-09-16T01:15:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.730+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 msgr2=0x55b97fd4c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.730+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55b97fd2c690 tx=0x7fd398014b90 comp rx=0 tx=0).stop 2024-09-16T01:15:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.732+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 shutdown_connections 2024-09-16T01:15:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.732+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd34cb0 0x55b97fd4cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.732+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd2a410 0x55b97fd4c9c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.732+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3181983680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd29af0 0x55b97fd32390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.732+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 >> 172.21.15.78:0/3181983680 conn(0x55b97fd26060 msgr2=0x55b97fd30bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.735+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 shutdown_connections 2024-09-16T01:15:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.737+0000 7fd3b2838740 1 -- 172.21.15.78:0/3181983680 wait complete. 2024-09-16T01:15:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.737+0000 7fd3b2838740 1 -- start start 2024-09-16T01:15:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.737+0000 7fd3b2838740 1 Processor -- start 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd38170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fdad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c00e510 con 0x55b97fd29af0 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c00f980 con 0x55b97fd2a410 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c00ebe0 con 0x55b97fd34cb0 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.739+0000 7fd3b2838740 10 client.0 _ll_register_callbacks cb 0x7fd3a4004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:15:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.740+0000 7fd3acf5f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.742+0000 7fd3acf5f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44282/0 (socket says 172.21.15.78:44282) 2024-09-16T01:15:54.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.742+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3652644055 learned_addr learned my addr 172.21.15.78:0/3652644055 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:15:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.743+0000 7fd3adf61640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fdad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3ad760640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd38170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3652644055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 msgr2=0x55b97fdad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fdad980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3652644055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 msgr2=0x55b97fd38170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd38170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.744+0000 7fd3acf5f640 1 -- 172.21.15.78:0/3652644055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b97fdadec0 con 0x55b97fd2a410 2024-09-16T01:15:54.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.747+0000 7fd3acf5f640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd398014640 tx=0x7fd398014670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f459977e7a88f7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:15:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.749+0000 7fd390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:15:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.749+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398033060 con 0x55b97fd2a410 2024-09-16T01:15:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.749+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd39800d950 con 0x55b97fd2a410 2024-09-16T01:15:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.749+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd398019040 con 0x55b97fd2a410 2024-09-16T01:15:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.754+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd39800db90 con 0x55b97fd2a410 2024-09-16T01:15:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.755+0000 7fd3b2838740 10 client.25457 fetch_fsmap learned FSMap version 31 2024-09-16T01:15:54.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.755+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b97fdaf0d0 con 0x55b97fd2a410 2024-09-16T01:15:54.757 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398019040 con 0x55b97fd2a410 2024-09-16T01:15:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd398013070 con 0x55b97fd2a410 2024-09-16T01:15:54.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 10 client.25457 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:15:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:15:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 20 client.25457 populate_metadata read hostname 'smithi078' 2024-09-16T01:15:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fdaf640 0x55b97fdb1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:15:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 4 client.25457 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:15:54.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.757+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b97fc95d70 con 0x55b97fdaf640 2024-09-16T01:15:54.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.763+0000 7fd3ad760640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fdaf640 0x55b97fdb1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:15:54.772 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.771+0000 7fd3ad760640 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fdaf640 0x55b97fdb1a60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fabaab23ff8cb7ba server_cookie=c676b3d01c58502 in_seq=0 out_seq=0 2024-09-16T01:15:54.773 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.772+0000 7fd390ff9640 10 client.25457 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:54 smithi078 ceph-mon[30937]: pgmap v156: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.9 MiB/s wr, 262 op/s 2024-09-16T01:15:54.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:54 smithi193 ceph-mon[35701]: pgmap v156: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.9 MiB/s wr, 262 op/s 2024-09-16T01:15:54.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:54 smithi148 ceph-mon[35030]: pgmap v156: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 2.9 MiB/s wr, 262 op/s 2024-09-16T01:15:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.901+0000 7fd390ff9640 1 -- 172.21.15.78:0/3652644055 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b97fc95d70 con 0x55b97fdaf640 2024-09-16T01:15:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.901+0000 7fd390ff9640 10 client.25457 handle_command_reply: tid=0 2024-09-16T01:15:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.901+0000 7fd3b2838740 1 client.25457 shutdown 2024-09-16T01:15:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 msgr2=0x55b97fd386b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd398014640 tx=0x7fd398014670 comp rx=0 tx=0).stop 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 shutdown_connections 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b97fdaf640 0x55b97fdb1a60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b97fd34cb0 0x55b97fdad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b97fd2a410 0x55b97fd386b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd390ff9640 0 client.25457 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 --2- 172.21.15.78:0/3652644055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b97fd29af0 0x55b97fd38170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 >> 172.21.15.78:0/3652644055 conn(0x55b97fd26060 msgr2=0x55b97fd323a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 shutdown_connections 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 1 -- 172.21.15.78:0/3652644055 wait complete. 2024-09-16T01:15:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.902+0000 7fd3b2838740 20 client.25457 trim_cache size 0 max 16384 2024-09-16T01:15:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:15:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:15:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:15:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:15:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 msgr2=0x7fd3940a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 0x7fd3940a5b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40045e0 tx=0x7fd3a40046c0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 msgr2=0x7fd3a81a7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd3a002aec0 tx=0x7fd3a0003f10 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 shutdown_connections 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd3940a3690 0x7fd3940a5b50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a811b140 0x7fd3a81aba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a81095d0 0x7fd3a81a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 --2- 172.21.15.78:0/529058932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3a81091f0 0x7fd3a81a7160 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 >> 172.21.15.78:0/529058932 conn(0x7fd3a806ce00 msgr2=0x7fd3a811b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.905+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 shutdown_connections 2024-09-16T01:15:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:15:54.906+0000 7fd3af9c4640 1 -- 172.21.15.78:0/529058932 wait complete. 2024-09-16T01:15:55.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:15:55.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:15:55.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:15:55.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:15:55.786 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:15:55.786 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:15:55.786 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:15:55.786 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:15:55.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:15:55.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:15:55.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:15:55.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:15:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 280.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:15:57.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:57 smithi078 ceph-mon[30937]: pgmap v157: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.4 MiB/s wr, 281 op/s 2024-09-16T01:15:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:57 smithi193 ceph-mon[35701]: pgmap v157: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.4 MiB/s wr, 281 op/s 2024-09-16T01:15:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:57 smithi148 ceph-mon[35030]: pgmap v157: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.4 MiB/s wr, 281 op/s 2024-09-16T01:15:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:15:59 smithi078 ceph-mon[30937]: pgmap v158: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 810 KiB/s wr, 60 op/s 2024-09-16T01:15:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:15:59 smithi193 ceph-mon[35701]: pgmap v158: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 810 KiB/s wr, 60 op/s 2024-09-16T01:15:59.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:15:59 smithi148 ceph-mon[35030]: pgmap v158: 129 pgs: 129 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 810 KiB/s wr, 60 op/s 2024-09-16T01:16:00.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 285.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:16:01.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:01 smithi078 ceph-mon[30937]: pgmap v159: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.8 MiB/s wr, 349 op/s 2024-09-16T01:16:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:01 smithi193 ceph-mon[35701]: pgmap v159: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.8 MiB/s wr, 349 op/s 2024-09-16T01:16:01.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:01 smithi148 ceph-mon[35030]: pgmap v159: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.8 MiB/s wr, 349 op/s 2024-09-16T01:16:02.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:16:02.909 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:03.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:03 smithi078 ceph-mon[30937]: pgmap v160: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.5 MiB/s wr, 359 op/s 2024-09-16T01:16:03.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 -- 172.21.15.78:0/2948376967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd741027a0 msgr2=0x7fdd741067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2948376967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd741027a0 0x7fdd741067f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009900 tx=0x7fdd5c02a410 comp rx=0 tx=0).stop 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 -- 172.21.15.78:0/2948376967 shutdown_connections 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2948376967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 0x7fdd7410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2948376967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd741027a0 0x7fdd741067f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2948376967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd74101df0 0x7fdd741021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 -- 172.21.15.78:0/2948376967 >> 172.21.15.78:0/2948376967 conn(0x7fdd740fd8b0 msgr2=0x7fdd740ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 -- 172.21.15.78:0/2948376967 shutdown_connections 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.413+0000 7fdd790dd640 1 -- 172.21.15.78:0/2948376967 wait complete. 2024-09-16T01:16:03.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 Processor -- start 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 -- start start 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 0x7fdd74198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 0x7fdd7419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74111fc0 con 0x7fdd74101df0 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74111e40 con 0x7fdd741027a0 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd790dd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74112140 con 0x7fdd74106f20 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:03.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42088/0 (socket says 172.21.15.78:42088) 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 -- 172.21.15.78:0/2924136136 learned_addr learned my addr 172.21.15.78:0/2924136136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd7359e640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 0x7fdd7419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd7259c640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 0x7fdd74198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 -- 172.21.15.78:0/2924136136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 msgr2=0x7fdd7419ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 0x7fdd7419ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 -- 172.21.15.78:0/2924136136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 msgr2=0x7fdd74198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 0x7fdd74198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.414+0000 7fdd72d9d640 1 -- 172.21.15.78:0/2924136136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7419d530 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd7259c640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 0x7fdd74198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd72d9d640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00ba50 tx=0x7fdd6c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c019070 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7419d820 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd6c00cbe0 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0076e0 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.415+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd741a5100 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.416+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd741021d0 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.417+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fdd6c01c070 con 0x7fdd74101df0 2024-09-16T01:16:03.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.418+0000 7fdd6bfff640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 0x7fdd600a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:03.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.418+0000 7fdd7259c640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 0x7fdd600a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:03.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.418+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fdd6c0cf550 con 0x7fdd74101df0 2024-09-16T01:16:03.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.418+0000 7fdd7259c640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 0x7fdd600a5870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c000c00 tx=0x7fdd5c032c70 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:03.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.422+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd6c09b660 con 0x7fdd74101df0 2024-09-16T01:16:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:03 smithi193 ceph-mon[35701]: pgmap v160: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.5 MiB/s wr, 359 op/s 2024-09-16T01:16:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.622+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd741998e0 con 0x7fdd74101df0 2024-09-16T01:16:03.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.623+0000 7fdd6bfff640 1 -- 172.21.15.78:0/2924136136 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fdd6c09f960 con 0x7fdd74101df0 2024-09-16T01:16:03.625 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:16:03.625 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:16:03.626 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 msgr2=0x7fdd600a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 0x7fdd600a5870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c000c00 tx=0x7fdd5c032c70 comp rx=0 tx=0).stop 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 msgr2=0x7fdd74198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00ba50 tx=0x7fdd6c00bf10 comp rx=0 tx=0).stop 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 shutdown_connections 2024-09-16T01:16:03.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd600a33b0 0x7fdd600a5870 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd74106f20 0x7fdd7419ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd741027a0 0x7fdd74198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 --2- 172.21.15.78:0/2924136136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd74101df0 0x7fdd74198580 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 >> 172.21.15.78:0/2924136136 conn(0x7fdd740fd8b0 msgr2=0x7fdd740feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 shutdown_connections 2024-09-16T01:16:03.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:03.625+0000 7fdd790dd640 1 -- 172.21.15.78:0/2924136136 wait complete. 2024-09-16T01:16:03.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:03 smithi148 ceph-mon[35030]: pgmap v160: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.5 MiB/s wr, 359 op/s 2024-09-16T01:16:04.360 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:04 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2924136136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:04.387 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 5be7c540 --release --await' 2024-09-16T01:16:04.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 5be7c540 --release --await 2024-09-16T01:16:04.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:04 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2924136136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:04.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:04 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2924136136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 -- 172.21.15.78:0/1879663533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 msgr2=0x7f18dc10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 --2- 172.21.15.78:0/1879663533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc10bb50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f18d0009900 tx=0x7f18d002a480 comp rx=0 tx=0).stop 2024-09-16T01:16:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 -- 172.21.15.78:0/1879663533 shutdown_connections 2024-09-16T01:16:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 --2- 172.21.15.78:0/1879663533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 --2- 172.21.15.78:0/1879663533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc10bb50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 --2- 172.21.15.78:0/1879663533 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 -- 172.21.15.78:0/1879663533 >> 172.21.15.78:0/1879663533 conn(0x7f18dc0fc7a0 msgr2=0x7f18dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 -- 172.21.15.78:0/1879663533 shutdown_connections 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.388+0000 7f18e0f82640 1 -- 172.21.15.78:0/1879663533 wait complete. 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 Processor -- start 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 -- start start 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc111f70 con 0x7f18dc103660 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc111df0 con 0x7f18dc102d40 2024-09-16T01:16:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18e0f82640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1120f0 con 0x7f18dc10c170 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18d9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18d9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45224/0 (socket says 172.21.15.78:45224) 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.389+0000 7f18d9d9b640 1 -- 172.21.15.78:0/2537213748 learned_addr learned my addr 172.21.15.78:0/2537213748 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18da59c640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18dad9d640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 -- 172.21.15.78:0/2537213748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 msgr2=0x7f18dc101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 -- 172.21.15.78:0/2537213748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 msgr2=0x7f18dc1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 -- 172.21.15.78:0/2537213748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc1a6890 con 0x7f18dc103660 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18dad9d640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18da59c640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18d9d9b640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006010 tx=0x7f18d00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18cb7fe640 1 -- 172.21.15.78:0/2537213748 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d003f030 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18dc1a6b20 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18cb7fe640 1 -- 172.21.15.78:0/2537213748 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18d0003ce0 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18cb7fe640 1 -- 172.21.15.78:0/2537213748 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d0035070 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.390+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18dc1a7050 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.392+0000 7f18cb7fe640 1 -- 172.21.15.78:0/2537213748 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f18d0042030 con 0x7f18dc103660 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 Processor -- start 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 -- start start 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e0c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18cb7fe640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 0x7f18c40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e19630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559569d71620 con 0x559569e0c980 2024-09-16T01:16:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559569d18e50 con 0x559569e0c060 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18e335b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559569d6d5e0 con 0x559569e17240 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18da59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18d9d9b640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 0x7f18c40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18da59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45228/0 (socket says 172.21.15.78:45228) 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.393+0000 7f18da59c640 1 -- 172.21.15.78:0/1839134903 learned_addr learned my addr 172.21.15.78:0/1839134903 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e0c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18cb7fe640 1 -- 172.21.15.78:0/2537213748 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f18d00f0740 con 0x7f18dc103660 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e19630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 -- 172.21.15.78:0/1839134903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 msgr2=0x559569e19630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e19630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 -- 172.21.15.78:0/1839134903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 msgr2=0x559569e0c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e0c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 -- 172.21.15.78:0/1839134903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559569d09d40 con 0x559569e0c980 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e19630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18d9d9b640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 0x7f18c40a5870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18d00b8bb0 tx=0x7f18d00b8a70 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18da59c640 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00bd40 tx=0x7f18cc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e62db77dee23982 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18c8ff9640 1 -- 172.21.15.78:0/1839134903 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc015070 con 0x559569e0c980 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18c8ff9640 1 -- 172.21.15.78:0/1839134903 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18cc00be30 con 0x559569e0c980 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.394+0000 7f18c8ff9640 1 -- 172.21.15.78:0/1839134903 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc011a30 con 0x559569e0c980 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- 172.21.15.78:0/1839134903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 msgr2=0x559569e16d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00bd40 tx=0x7f18cc00f730 comp rx=0 tx=0).stop 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- 172.21.15.78:0/1839134903 shutdown_connections 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e19630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e16d00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 --2- 172.21.15.78:0/1839134903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e0c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- 172.21.15.78:0/1839134903 >> 172.21.15.78:0/1839134903 conn(0x559569e085f0 msgr2=0x559569e089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- 172.21.15.78:0/1839134903 shutdown_connections 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- 172.21.15.78:0/1839134903 wait complete. 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.395+0000 7f18e335b740 1 -- start start 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 Processor -- start 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e14180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e14c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559569e19e10 con 0x559569e0c980 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559569cf9f10 con 0x559569e0c060 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559569e19f90 con 0x559569e17240 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18e335b740 10 client.0 _ll_register_callbacks cb 0x559569d1ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45244/0 (socket says 172.21.15.78:45244) 2024-09-16T01:16:05.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 -- 172.21.15.78:0/3023258079 learned_addr learned my addr 172.21.15.78:0/3023258079 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18dad9d640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e14180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18dad9d640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e14c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 -- 172.21.15.78:0/3023258079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 msgr2=0x559569e14c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e14c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 -- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 msgr2=0x559569e14180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e14180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 -- 172.21.15.78:0/3023258079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559569a80580 con 0x559569e0c980 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18dad9d640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e14180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18da59c640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00c010 tx=0x7f18cc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59a53b21773a2a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.396+0000 7f18c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:05.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc015070 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f18cc005310 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18cc00be30 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18cc005510 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 10 client.16023 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559569b4a960 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18cc00be30 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f18cc004b30 con 0x559569e0c980 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 10 client.16023 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 10 client.16023 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 20 client.16023 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:05.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e26e30 0x559569e29220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 4 client.16023 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559569d7eb90 con 0x559569e26e30 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.397+0000 7f18dad9d640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e26e30 0x559569e29220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18dad9d640 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e26e30 0x559569e29220 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d8656a2073ef58 server_cookie=820d1626ca206b92 in_seq=0 out_seq=0 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18c1ffb640 10 client.16023 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18c1ffb640 1 -- 172.21.15.78:0/3023258079 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559569d7eb90 con 0x559569e26e30 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18c1ffb640 10 client.16023 handle_command_reply: tid=0 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18e335b740 1 client.16023 shutdown 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 msgr2=0x559569e146c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00c010 tx=0x7f18cc00bab0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 shutdown_connections 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.398+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e26e30 0x559569e29220 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e17240 0x559569e14c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c980 0x559569e146c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 --2- 172.21.15.78:0/3023258079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c060 0x559569e14180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 >> 172.21.15.78:0/3023258079 conn(0x559569e085f0 msgr2=0x559569e07750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18c1ffb640 0 client.16023 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 shutdown_connections 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 1 -- 172.21.15.78:0/3023258079 wait complete. 2024-09-16T01:16:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.399+0000 7f18e335b740 20 client.16023 trim_cache size 0 max 16384 2024-09-16T01:16:05.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.415+0000 7f18e335b740 1 Processor -- start 2024-09-16T01:16:05.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.415+0000 7f18e335b740 1 -- start start 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.415+0000 7f18e335b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18e335b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c980 0x559569e2e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18e335b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e17240 0x559569e2eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18e335b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1b3af0 con 0x559569e0c060 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18e335b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1b13c0 con 0x559569e17240 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18e335b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1af210 con 0x559569e0c980 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45256/0 (socket says 172.21.15.78:45256) 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 -- 172.21.15.78:0/1938674654 learned_addr learned my addr 172.21.15.78:0/1938674654 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18da59c640 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c980 0x559569e2e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 -- 172.21.15.78:0/1938674654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c980 msgr2=0x559569e2e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c980 0x559569e2e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 -- 172.21.15.78:0/1938674654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e17240 msgr2=0x559569e2eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e17240 0x559569e2eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 -- 172.21.15.78:0/1938674654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559569e09780 con 0x559569e0c060 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18dad9d640 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007310 tx=0x7f18d4015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b313ed72f65e558 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18c3fff640 1 -- 172.21.15.78:0/1938674654 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4004d20 con 0x559569e0c060 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18c3fff640 1 -- 172.21.15.78:0/1938674654 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18d40119e0 con 0x559569e0c060 2024-09-16T01:16:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.416+0000 7f18c3fff640 1 -- 172.21.15.78:0/1938674654 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d40055c0 con 0x559569e0c060 2024-09-16T01:16:05.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- 172.21.15.78:0/1938674654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 msgr2=0x559569e0e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007310 tx=0x7f18d4015040 comp rx=0 tx=0).stop 2024-09-16T01:16:05.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- 172.21.15.78:0/1938674654 shutdown_connections 2024-09-16T01:16:05.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e17240 0x559569e2eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c980 0x559569e2e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 --2- 172.21.15.78:0/1938674654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e0c060 0x559569e0e1e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- 172.21.15.78:0/1938674654 >> 172.21.15.78:0/1938674654 conn(0x559569e085f0 msgr2=0x559569e0dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- 172.21.15.78:0/1938674654 shutdown_connections 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- 172.21.15.78:0/1938674654 wait complete. 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 -- start start 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.417+0000 7f18e335b740 1 Processor -- start 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c060 0x559569e1ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c980 0x559569e1bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1af880 con 0x559569e17240 2024-09-16T01:16:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1b0cf0 con 0x559569e0c980 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc1aff50 con 0x559569e0c060 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 10 client.0 _ll_register_callbacks cb 0x7f18cc00ff60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45266/0 (socket says 172.21.15.78:45266) 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 -- 172.21.15.78:0/500223523 learned_addr learned my addr 172.21.15.78:0/500223523 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c060 0x559569e1ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18da59c640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c980 0x559569e1bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 -- 172.21.15.78:0/500223523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c060 msgr2=0x559569e1ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c060 0x559569e1ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 -- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c980 msgr2=0x559569e1bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c980 0x559569e1bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 -- 172.21.15.78:0/500223523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559569e89f90 con 0x559569e17240 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18dad9d640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007d50 tx=0x7f18d40051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab29eb19eda6646 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4004e50 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f18d4005240 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.418+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18d40119e0 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18d403c050 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 10 client.16029 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559569e924f0 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18d40119e0 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f18d4024030 con 0x559569e17240 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 10 client.16029 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 10 client.16029 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 20 client.16029 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e92ac0 0x559569e94ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 4 client.16029 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5be7c540", "release": true, "await": true}] 2024-09-16T01:16:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5be7c540", "release": true, "await": true}) -- 0x559569d790a0 con 0x559569e92ac0 2024-09-16T01:16:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.419+0000 7f18da59c640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e92ac0 0x559569e94ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.420+0000 7f18da59c640 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e92ac0 0x559569e94ee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d303661dbc4a064 server_cookie=82467089b3f5356b in_seq=0 out_seq=0 2024-09-16T01:16:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.420+0000 7f18c1ffb640 10 client.16029 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18c1ffb640 1 -- 172.21.15.78:0/500223523 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559569d790a0 con 0x559569e92ac0 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18c1ffb640 10 client.16029 handle_command_reply: tid=0 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 client.16029 shutdown 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 msgr2=0x559569e898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f18d4007d50 tx=0x7f18d40051a0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 shutdown_connections 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559569e92ac0 0x559569e94ee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559569e17240 0x559569e898f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559569e0c980 0x559569e1bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 --2- 172.21.15.78:0/500223523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559569e0c060 0x559569e1ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18c1ffb640 0 client.16029 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:05.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.468+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 >> 172.21.15.78:0/500223523 conn(0x559569e085f0 msgr2=0x559569e07750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.469+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 shutdown_connections 2024-09-16T01:16:05.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.470+0000 7f18e335b740 1 -- 172.21.15.78:0/500223523 wait complete. 2024-09-16T01:16:05.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:05 smithi078 ceph-mon[30937]: pgmap v161: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.5 MiB/s wr, 361 op/s 2024-09-16T01:16:05.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.473+0000 7f18e335b740 20 client.16029 trim_cache size 0 max 16384 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 20, 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "5be7c540": { 2024-09-16T01:16:05.478 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 20, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 145.3, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:16:05.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:05.480 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:16:05.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 msgr2=0x7f18c40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 0x7f18c40a5870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18d00b8bb0 tx=0x7f18d00b8a70 comp rx=0 tx=0).stop 2024-09-16T01:16:05.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 msgr2=0x7f18dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006010 tx=0x7f18d00042f0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 shutdown_connections 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f18c40a33b0 0x7f18c40a5870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18dc10c170 0x7f18dc101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18dc103660 0x7f18dc100ee0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 --2- 172.21.15.78:0/2537213748 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18dc102d40 0x7f18dc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.476+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 >> 172.21.15.78:0/2537213748 conn(0x7f18dc0fc7a0 msgr2=0x7f18dc0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.477+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 shutdown_connections 2024-09-16T01:16:05.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:05.477+0000 7f18e0f82640 1 -- 172.21.15.78:0/2537213748 wait complete. 2024-09-16T01:16:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:05 smithi193 ceph-mon[35701]: pgmap v161: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.5 MiB/s wr, 361 op/s 2024-09-16T01:16:05.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:05 smithi148 ceph-mon[35030]: pgmap v161: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.5 MiB/s wr, 361 op/s 2024-09-16T01:16:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 290.0 s, 18.6 tps, lat 16137.085 ms stddev 11308.584 2024-09-16T01:16:06.691 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5be7c540', total seconds elapsed: 29.4 2024-09-16T01:16:06.691 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-16T01:16:06.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:16:07.144 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:07.282 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:07 smithi078 ceph-mon[30937]: pgmap v162: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-09-16T01:16:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:07 smithi193 ceph-mon[35701]: pgmap v162: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-09-16T01:16:07.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:07 smithi148 ceph-mon[35030]: pgmap v162: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.1 MiB/s wr, 422 op/s 2024-09-16T01:16:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 -- 172.21.15.78:0/1551727854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 msgr2=0x7f400c1056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 --2- 172.21.15.78:0/1551727854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c1056b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009900 tx=0x7f3ffc02a410 comp rx=0 tx=0).stop 2024-09-16T01:16:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 -- 172.21.15.78:0/1551727854 shutdown_connections 2024-09-16T01:16:08.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 --2- 172.21.15.78:0/1551727854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400c105de0 0x7f400c10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 --2- 172.21.15.78:0/1551727854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c1056b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 --2- 172.21.15.78:0/1551727854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f400c101470 0x7f400c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 -- 172.21.15.78:0/1551727854 >> 172.21.15.78:0/1551727854 conn(0x7f400c077230 msgr2=0x7f400c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 -- 172.21.15.78:0/1551727854 shutdown_connections 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.730+0000 7f40134d3640 1 -- 172.21.15.78:0/1551727854 wait complete. 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 Processor -- start 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 -- start start 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400c101470 0x7f400c197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f400c105de0 0x7f400c19bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f400c110e00 con 0x7f400c101470 2024-09-16T01:16:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f400c110c80 con 0x7f400c105de0 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.732+0000 7f40134d3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f400c110f80 con 0x7f400c101e20 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54460/0 (socket says 172.21.15.78:54460) 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 -- 172.21.15.78:0/3274105671 learned_addr learned my addr 172.21.15.78:0/3274105671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 -- 172.21.15.78:0/3274105671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f400c105de0 msgr2=0x7f400c19bbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f400c105de0 0x7f400c19bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 -- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400c101470 msgr2=0x7f400c197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400c101470 0x7f400c197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.733+0000 7f4010a6e640 1 -- 172.21.15.78:0/3274105671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c19c2d0 con 0x7f400c101e20 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f4010a6e640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc006010 tx=0x7f3ffc004ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc044070 con 0x7f400c101e20 2024-09-16T01:16:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400c19c560 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400c100100 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc005510 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.734+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc040ba0 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.736+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ff0005290 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.737+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3ffc047070 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.738+0000 7f40027fc640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 0x7f3ff80a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.739+0000 7f401126f640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 0x7f3ff80a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.739+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f3ffc0f4ab0 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.748+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ffc0c1130 con 0x7f400c101e20 2024-09-16T01:16:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:07.751+0000 7f401126f640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 0x7f3ff80a5a40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f40080058c0 tx=0x7f4008005850 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 -- 172.21.15.78:0/3130594910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde501091f0 msgr2=0x7fde50110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 --2- 172.21.15.78:0/3130594910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde501091f0 0x7fde50110250 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde48009930 tx=0x7fde4802a480 comp rx=0 tx=0).stop 2024-09-16T01:16:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 -- 172.21.15.78:0/3130594910 shutdown_connections 2024-09-16T01:16:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 --2- 172.21.15.78:0/3130594910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde5011b140 0x7fde5011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 --2- 172.21.15.78:0/3130594910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 0x7fde50109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 --2- 172.21.15.78:0/3130594910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde501091f0 0x7fde50110250 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.054+0000 7fde57bfd640 1 -- 172.21.15.78:0/3130594910 >> 172.21.15.78:0/3130594910 conn(0x7fde5006ce00 msgr2=0x7fde5006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.058+0000 7fde57bfd640 1 -- 172.21.15.78:0/3130594910 shutdown_connections 2024-09-16T01:16:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.060+0000 7fde57bfd640 1 -- 172.21.15.78:0/3130594910 wait complete. 2024-09-16T01:16:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.060+0000 7fde57bfd640 1 Processor -- start 2024-09-16T01:16:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.060+0000 7fde57bfd640 1 -- start start 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 0x7fde501a7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde5011b140 0x7fde501abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde50120db0 con 0x7fde501091f0 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde50120c30 con 0x7fde5011b140 2024-09-16T01:16:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde57bfd640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde50120f30 con 0x7fde501095d0 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde55999640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde55999640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45292/0 (socket says 172.21.15.78:45292) 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde55999640 1 -- 172.21.15.78:0/1262373517 learned_addr learned my addr 172.21.15.78:0/1262373517 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde5619a640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde5011b140 0x7fde501abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.061+0000 7fde55198640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 0x7fde501a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.062+0000 7fde55999640 1 -- 172.21.15.78:0/1262373517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 msgr2=0x7fde501a7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.062+0000 7fde55999640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 0x7fde501a7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.062+0000 7fde55999640 1 -- 172.21.15.78:0/1262373517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde5011b140 msgr2=0x7fde501abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.062+0000 7fde55999640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde5011b140 0x7fde501abaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.062+0000 7fde55999640 1 -- 172.21.15.78:0/1262373517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde501ac1d0 con 0x7fde501091f0 2024-09-16T01:16:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.064+0000 7fde55999640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fde4802a9a0 tx=0x7fde48005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.065+0000 7fde46ffd640 1 -- 172.21.15.78:0/1262373517 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48040030 con 0x7fde501091f0 2024-09-16T01:16:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.065+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde501ac460 con 0x7fde501091f0 2024-09-16T01:16:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.065+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde501b3d50 con 0x7fde501091f0 2024-09-16T01:16:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.069+0000 7fde5aa7b740 1 Processor -- start 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.069+0000 7fde46ffd640 1 -- 172.21.15.78:0/1262373517 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde48003df0 con 0x7fde501091f0 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.069+0000 7fde46ffd640 1 -- 172.21.15.78:0/1262373517 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4803ca70 con 0x7fde501091f0 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 -- start start 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd61a2f0 con 0x55ebdd6b61f0 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd5c2270 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.070+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd6162b0 con 0x55ebdd6b58d0 2024-09-16T01:16:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde46ffd640 1 -- 172.21.15.78:0/1262373517 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fde48043030 con 0x7fde501091f0 2024-09-16T01:16:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde5619a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:55472/0 (socket says 172.21.15.78:55472) 2024-09-16T01:16:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 -- 172.21.15.78:0/3601283422 learned_addr learned my addr 172.21.15.78:0/3601283422 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 -- 172.21.15.78:0/3601283422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 msgr2=0x55ebdd6b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 -- 172.21.15.78:0/3601283422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 msgr2=0x55ebdd6d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.073+0000 7fde55198640 1 -- 172.21.15.78:0/3601283422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdd5c1340 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.074+0000 7fde46ffd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 0x7fde3c0a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.074+0000 7fde46ffd640 1 -- 172.21.15.78:0/1262373517 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fde480f0c90 con 0x7fde501091f0 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.075+0000 7fde55198640 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00a480 tx=0x7fde4c0096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df4e0cbefd18e1c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.075+0000 7fde5619a640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 0x7fde3c0a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.075+0000 7fde3bfff640 1 -- 172.21.15.78:0/3601283422 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c013790 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.075+0000 7fde3bfff640 1 -- 172.21.15.78:0/3601283422 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde4c019070 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.075+0000 7fde3bfff640 1 -- 172.21.15.78:0/3601283422 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c014e50 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.076+0000 7fde5aa7b740 1 -- 172.21.15.78:0/3601283422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 msgr2=0x55ebdd6d40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.076+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00a480 tx=0x7fde4c0096d0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.078+0000 7fde5aa7b740 1 -- 172.21.15.78:0/3601283422 shutdown_connections 2024-09-16T01:16:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.078+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6d40b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.078+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.078+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/3601283422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.078+0000 7fde5aa7b740 1 -- 172.21.15.78:0/3601283422 >> 172.21.15.78:0/3601283422 conn(0x55ebdd6b2060 msgr2=0x55ebdd6b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.081+0000 7fde5619a640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 0x7fde3c0a5b00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde40009a70 tx=0x7fde400090e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.082+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ff0005850 con 0x7f400c101e20 2024-09-16T01:16:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.081+0000 7fde5aa7b740 1 -- 172.21.15.78:0/3601283422 shutdown_connections 2024-09-16T01:16:08.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.083+0000 7f40027fc640 1 -- 172.21.15.78:0/3274105671 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f3ffc0c0a30 con 0x7f400c101e20 2024-09-16T01:16:08.086 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:16:08.086 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:16:08.087 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:16:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.084+0000 7fde5aa7b740 1 -- 172.21.15.78:0/3601283422 wait complete. 2024-09-16T01:16:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.084+0000 7fde5aa7b740 1 -- start start 2024-09-16T01:16:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.084+0000 7fde5aa7b740 1 Processor -- start 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6be420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd6b8a10 con 0x55ebdd6b58d0 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd5a3290 con 0x55ebdd6b61f0 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebdd6b8b90 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.085+0000 7fde5aa7b740 10 client.0 _ll_register_callbacks cb 0x55ebdd5d4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.086+0000 7fde55999640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.087+0000 7fde55999640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45304/0 (socket says 172.21.15.78:45304) 2024-09-16T01:16:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.087+0000 7fde55999640 1 -- 172.21.15.78:0/229146303 learned_addr learned my addr 172.21.15.78:0/229146303 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.087+0000 7fde55198640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.087+0000 7fde5619a640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.088+0000 7fde55999640 1 -- 172.21.15.78:0/229146303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 msgr2=0x55ebdd6be420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.088+0000 7fde55999640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6be420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.088+0000 7fde55999640 1 -- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 msgr2=0x55ebdd6c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.088+0000 7fde55999640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.088+0000 7fde55999640 1 -- 172.21.15.78:0/229146303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdd32d210 con 0x55ebdd6b58d0 2024-09-16T01:16:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.089+0000 7fde55198640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.089+0000 7fde5619a640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde55999640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fde48004650 tx=0x7fde480bc590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e177ec2b8d2175 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde397fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4804a040 con 0x55ebdd6b58d0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde4c019070 con 0x55ebdd6b58d0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fde480ee270 con 0x55ebdd6b58d0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.090+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde480ee470 con 0x55ebdd6b58d0 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.091+0000 7fde5aa7b740 10 client.16038 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.091+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebdd3f9620 con 0x55ebdd6b58d0 2024-09-16T01:16:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.091+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde4c019070 con 0x55ebdd6b58d0 2024-09-16T01:16:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fde4803cc10 con 0x55ebdd6b58d0 2024-09-16T01:16:08.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 msgr2=0x7f3ff80a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 0x7f3ff80a5a40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f40080058c0 tx=0x7f4008005850 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 msgr2=0x7f400c197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc006010 tx=0x7f3ffc004ed0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 shutdown_connections 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff80a3580 0x7f3ff80a5a40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f400c105de0 0x7f400c19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400c101e20 0x7f400c197840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 --2- 172.21.15.78:0/3274105671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400c101470 0x7f400c197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 >> 172.21.15.78:0/3274105671 conn(0x7f400c077230 msgr2=0x7f400c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 shutdown_connections 2024-09-16T01:16:08.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:08.092+0000 7f40134d3640 1 -- 172.21.15.78:0/3274105671 wait complete. 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 10 client.16038 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 20 client.16038 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd6ce640 0x55ebdd6d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 4 client.16038 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.092+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebdd627a20 con 0x55ebdd6ce640 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.093+0000 7fde55198640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd6ce640 0x55ebdd6d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.093+0000 7fde55198640 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd6ce640 0x55ebdd6d0a70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d955e93ac9320b7a server_cookie=788ee88c7a50e205 in_seq=0 out_seq=0 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.093+0000 7fde397fa640 10 client.16038 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde397fa640 1 -- 172.21.15.78:0/229146303 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ebdd627a20 con 0x55ebdd6ce640 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde397fa640 10 client.16038 handle_command_reply: tid=0 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 client.16038 shutdown 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 msgr2=0x55ebdd6c30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fde48004650 tx=0x7fde480bc590 comp rx=0 tx=0).stop 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 shutdown_connections 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd6ce640 0x55ebdd6d0a70 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/229146303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c30d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 >> 172.21.15.78:0/229146303 conn(0x55ebdd6b2060 msgr2=0x55ebdd6b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.094+0000 7fde397fa640 0 client.16038 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.096+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 shutdown_connections 2024-09-16T01:16:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.101+0000 7fde5aa7b740 1 -- 172.21.15.78:0/229146303 wait complete. 2024-09-16T01:16:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.101+0000 7fde5aa7b740 20 client.16038 trim_cache size 0 max 16384 2024-09-16T01:16:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 Processor -- start 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 -- start start 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6fb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde340126f0 con 0x55ebdd6b61f0 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde3400ffc0 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.136+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde3400dec0 con 0x55ebdd6b58d0 2024-09-16T01:16:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55198640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6fb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55198640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6fb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45312/0 (socket says 172.21.15.78:45312) 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55198640 1 -- 172.21.15.78:0/612562244 learned_addr learned my addr 172.21.15.78:0/612562244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54510/0 (socket says 172.21.15.78:54510) 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde5619a640 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 -- 172.21.15.78:0/612562244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 msgr2=0x55ebdd6fbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6fbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 -- 172.21.15.78:0/612562244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 msgr2=0x55ebdd6fb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6fb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 -- 172.21.15.78:0/612562244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebdd6be790 con 0x55ebdd6b58d0 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde55999640 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fde48006010 tx=0x7fde480bc6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb5a10f81ebb933c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde3b7fe640 1 -- 172.21.15.78:0/612562244 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480baf30 con 0x55ebdd6b58d0 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde3b7fe640 1 -- 172.21.15.78:0/612562244 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde4803cc10 con 0x55ebdd6b58d0 2024-09-16T01:16:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.137+0000 7fde3b7fe640 1 -- 172.21.15.78:0/612562244 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480bce90 con 0x55ebdd6b58d0 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.138+0000 7fde5aa7b740 1 -- 172.21.15.78:0/612562244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 msgr2=0x55ebdd6d1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.138+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fde48006010 tx=0x7fde480bc6c0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.138+0000 7fde5aa7b740 1 -- 172.21.15.78:0/612562244 shutdown_connections 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.138+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6fbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.138+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6fb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.139+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/612562244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6d1310 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.139+0000 7fde5aa7b740 1 -- 172.21.15.78:0/612562244 >> 172.21.15.78:0/612562244 conn(0x55ebdd6b2060 msgr2=0x55ebdd6bd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.139+0000 7fde5aa7b740 1 -- 172.21.15.78:0/612562244 shutdown_connections 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.139+0000 7fde5aa7b740 1 -- 172.21.15.78:0/612562244 wait complete. 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.140+0000 7fde5aa7b740 1 -- start start 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.140+0000 7fde5aa7b740 1 Processor -- start 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde3400e480 con 0x55ebdd6b61f0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde3400f8f0 con 0x55ebdd6d1cc0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde3400eb50 con 0x55ebdd6b58d0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5aa7b740 10 client.0 _ll_register_callbacks cb 0x7fde480bca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5619a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5619a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:55492/0 (socket says 172.21.15.78:55492) 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde5619a640 1 -- 172.21.15.78:0/606182473 learned_addr learned my addr 172.21.15.78:0/606182473 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55198640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 -- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 msgr2=0x55ebdd6c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 -- 172.21.15.78:0/606182473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 msgr2=0x55ebdd6c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.141+0000 7fde55999640 1 -- 172.21.15.78:0/606182473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebdd6c9480 con 0x55ebdd6b58d0 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde5619a640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde55999640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c4840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde480043a0 tx=0x7fde4802a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97ee9ecf066a08f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde397fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:16:08.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480bd090 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fde4802ac40 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde4803cc10 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde4802ae40 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde5aa7b740 10 client.25525 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ebdd6c9480 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.142+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde4803cc10 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fde480bc080 con 0x55ebdd6b58d0 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 10 client.25525 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 10 client.25525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 20 client.25525 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd75d2b0 0x55ebdd75f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 4 client.25525 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebdd621d70 con 0x55ebdd75d2b0 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.143+0000 7fde55198640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd75d2b0 0x55ebdd75f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.144+0000 7fde397fa640 10 client.25525 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:08.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.144+0000 7fde55198640 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd75d2b0 0x55ebdd75f6a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b16dcb527b51c0 server_cookie=2eb462be4eacb9c6 in_seq=0 out_seq=0 2024-09-16T01:16:08.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:08 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3274105671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.289+0000 7fde397fa640 1 -- 172.21.15.78:0/606182473 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ebdd621d70 con 0x55ebdd75d2b0 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.289+0000 7fde397fa640 10 client.25525 handle_command_reply: tid=0 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.289+0000 7fde5aa7b740 1 client.25525 shutdown 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.289+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 msgr2=0x55ebdd6c4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c4840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde480043a0 tx=0x7fde4802a480 comp rx=0 tx=0).stop 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 shutdown_connections 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ebdd75d2b0 0x55ebdd75f6a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde397fa640 0 client.25525 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ebdd6d1cc0 0x55ebdd6c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebdd6b61f0 0x55ebdd6c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 --2- 172.21.15.78:0/606182473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebdd6b58d0 0x55ebdd6c4840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 >> 172.21.15.78:0/606182473 conn(0x55ebdd6b2060 msgr2=0x55ebdd6bd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 shutdown_connections 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.290+0000 7fde5aa7b740 1 -- 172.21.15.78:0/606182473 wait complete. 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.291+0000 7fde5aa7b740 20 client.25525 trim_cache size 0 max 16384 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:16:08.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:16:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.291+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 msgr2=0x7fde3c0a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.291+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 0x7fde3c0a5b00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde40009a70 tx=0x7fde400090e0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.291+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 msgr2=0x7fde501a71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.291+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fde4802a9a0 tx=0x7fde48005230 comp rx=0 tx=0).stop 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 shutdown_connections 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde3c0a3640 0x7fde3c0a5b00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde5011b140 0x7fde501abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde501095d0 0x7fde501a7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 --2- 172.21.15.78:0/1262373517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde501091f0 0x7fde501a71e0 secure :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fde4802a9a0 tx=0x7fde48005230 comp rx=0 tx=0).stop 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 >> 172.21.15.78:0/1262373517 conn(0x7fde5006ce00 msgr2=0x7fde5011b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 shutdown_connections 2024-09-16T01:16:08.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:08.292+0000 7fde57bfd640 1 -- 172.21.15.78:0/1262373517 wait complete. 2024-09-16T01:16:08.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:08 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3274105671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:08.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:08 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3274105671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:09.301 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-16T01:16:09.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-16T01:16:09.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:09 smithi078 ceph-mon[30937]: pgmap v163: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 399 op/s 2024-09-16T01:16:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:09 smithi193 ceph-mon[35701]: pgmap v163: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 399 op/s 2024-09-16T01:16:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:09 smithi148 ceph-mon[35030]: pgmap v163: 129 pgs: 129 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 399 op/s 2024-09-16T01:16:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:10.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.477+0000 7f33b47f3640 1 -- 172.21.15.78:0/3142217995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac100a50 msgr2=0x7f33ac1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.477+0000 7f33b47f3640 1 --2- 172.21.15.78:0/3142217995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac100a50 0x7f33ac1057e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f339c009be0 tx=0x7f339c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.478+0000 7f33b47f3640 1 -- 172.21.15.78:0/3142217995 shutdown_connections 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.478+0000 7f33b47f3640 1 --2- 172.21.15.78:0/3142217995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac100a50 0x7f33ac1057e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.478+0000 7f33b47f3640 1 --2- 172.21.15.78:0/3142217995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33ac100090 0x7f33ac100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.478+0000 7f33b47f3640 1 --2- 172.21.15.78:0/3142217995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac106b70 0x7f33ac106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.478+0000 7f33b47f3640 1 -- 172.21.15.78:0/3142217995 >> 172.21.15.78:0/3142217995 conn(0x7f33ac077220 msgr2=0x7f33ac0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.480+0000 7f33b47f3640 1 -- 172.21.15.78:0/3142217995 shutdown_connections 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.483+0000 7f33b47f3640 1 -- 172.21.15.78:0/3142217995 wait complete. 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.484+0000 7f33b47f3640 1 Processor -- start 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.484+0000 7f33b47f3640 1 -- start start 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33ac100090 0x7f33ac073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac106b70 0x7f33ac06cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac1185c0 con 0x7f33ac106b70 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac118440 con 0x7f33ac100a50 2024-09-16T01:16:10.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.485+0000 7f33b47f3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac118740 con 0x7f33ac100090 2024-09-16T01:16:10.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.487+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.487+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:55498/0 (socket says 172.21.15.78:55498) 2024-09-16T01:16:10.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.487+0000 7f33b1d8e640 1 -- 172.21.15.78:0/1747159839 learned_addr learned my addr 172.21.15.78:0/1747159839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:10.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.487+0000 7f33b2d90640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac106b70 0x7f33ac06cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.490+0000 7f33b1d8e640 1 -- 172.21.15.78:0/1747159839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33ac100090 msgr2=0x7f33ac073910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:16:10.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.490+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33ac100090 0x7f33ac073910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.490+0000 7f33b1d8e640 1 -- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac106b70 msgr2=0x7f33ac06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.490+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac106b70 0x7f33ac06cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.490+0000 7f33b1d8e640 1 -- 172.21.15.78:0/1747159839 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac06d3b0 con 0x7f33ac100a50 2024-09-16T01:16:10.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.493+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f33a000e9c0 tx=0x7f33a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.497+0000 7f33ab7fe640 1 -- 172.21.15.78:0/1747159839 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a000ccc0 con 0x7f33ac100a50 2024-09-16T01:16:10.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.497+0000 7f33ab7fe640 1 -- 172.21.15.78:0/1747159839 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a0005500 con 0x7f33ac100a50 2024-09-16T01:16:10.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.497+0000 7f33ab7fe640 1 -- 172.21.15.78:0/1747159839 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0010430 con 0x7f33ac100a50 2024-09-16T01:16:10.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.497+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac06d6a0 con 0x7f33ac100a50 2024-09-16T01:16:10.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.497+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac071670 con 0x7f33ac100a50 2024-09-16T01:16:10.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.502+0000 7f33b6bcc740 1 Processor -- start 2024-09-16T01:16:10.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 -- start start 2024-09-16T01:16:10.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b934850 0x55f22b934c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b941e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f22b899160 con 0x55f22b93fa30 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f22b840720 con 0x55f22b934850 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.504+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f22b895120 con 0x55f22b935170 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.505+0000 7f33b2d90640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b934850 0x55f22b934c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.506+0000 7f33b258f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.506+0000 7f33b258f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54540/0 (socket says 172.21.15.78:54540) 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.506+0000 7f33b258f640 1 -- 172.21.15.78:0/559142577 learned_addr learned my addr 172.21.15.78:0/559142577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.507+0000 7f33b258f640 1 -- 172.21.15.78:0/559142577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b934850 msgr2=0x55f22b934c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.507+0000 7f33b258f640 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b934850 0x55f22b934c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.507+0000 7f33b258f640 1 -- 172.21.15.78:0/559142577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 msgr2=0x55f22b941e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.507+0000 7f33b258f640 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b941e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.507+0000 7f33b258f640 1 -- 172.21.15.78:0/559142577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22b60aba0 con 0x55f22b935170 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.508+0000 7f33b258f640 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33a400ba00 tx=0x7f33a400bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d08e8f5f3cf7e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.508+0000 7f33a8ff9640 1 -- 172.21.15.78:0/559142577 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4016070 con 0x55f22b935170 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.508+0000 7f33ab7fe640 1 -- 172.21.15.78:0/1747159839 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f33a0005020 con 0x7f33ac100a50 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.508+0000 7f33a8ff9640 1 -- 172.21.15.78:0/559142577 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a4005500 con 0x55f22b935170 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.508+0000 7f33a8ff9640 1 -- 172.21.15.78:0/559142577 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4010460 con 0x55f22b935170 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.509+0000 7f33ab7fe640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 0x7f33940a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.509+0000 7f33ab7fe640 1 -- 172.21.15.78:0/1747159839 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33a00d01e0 con 0x7f33ac100a50 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b2d90640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 0x7f33940a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 -- 172.21.15.78:0/559142577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 msgr2=0x55f22b93f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33a400ba00 tx=0x7f33a400bec0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b2d90640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 0x7f33940a5a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f339c006010 tx=0x7f339c0098e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 -- 172.21.15.78:0/559142577 shutdown_connections 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b941e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b93f4f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/559142577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b934850 0x55f22b934c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.510+0000 7f33b6bcc740 1 -- 172.21.15.78:0/559142577 >> 172.21.15.78:0/559142577 conn(0x55f22b930de0 msgr2=0x55f22b9311c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.511+0000 7f33b6bcc740 1 -- 172.21.15.78:0/559142577 shutdown_connections 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.511+0000 7f33b6bcc740 1 -- 172.21.15.78:0/559142577 wait complete. 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.511+0000 7f33b6bcc740 1 -- start start 2024-09-16T01:16:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.511+0000 7f33b6bcc740 1 Processor -- start 2024-09-16T01:16:10.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b93c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b93fa30 0x55f22b93d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f22b942600 con 0x55f22b934850 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f22b821900 con 0x55f22b935170 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f22b942780 con 0x55f22b93fa30 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b6bcc740 10 client.0 _ll_register_callbacks cb 0x55f22b81f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:55512/0 (socket says 172.21.15.78:55512) 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b1d8e640 1 -- 172.21.15.78:0/1148997974 learned_addr learned my addr 172.21.15.78:0/1148997974 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b2d90640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b93fa30 0x55f22b93d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.512+0000 7f33b258f640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b93c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 -- 172.21.15.78:0/1148997974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b93fa30 msgr2=0x55f22b93d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b93fa30 0x55f22b93d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 -- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 msgr2=0x55f22b93cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 -- 172.21.15.78:0/1148997974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22b861c50 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f33b258f640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b93c840 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f33a400ba00 tx=0x7f33a4005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d8832691b56352 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f3391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.513+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4016030 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33a4004450 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33a4005500 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33a40046c0 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 10 client.16053 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f22b94e280 con 0x55f22b934850 2024-09-16T01:16:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a4005500 con 0x55f22b934850 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33a4022070 con 0x55f22b934850 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 10 client.16053 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 10 client.16053 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 20 client.16053 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b94f630 0x55f22b951a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.514+0000 7f33b6bcc740 4 client.16053 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.515+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f22b8a6890 con 0x55f22b94f630 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.515+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b94f630 0x55f22b951a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.515+0000 7f3391ffb640 10 client.16053 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.515+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b94f630 0x55f22b951a20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54189da5e7cae0d8 server_cookie=936f2d5582583dfd in_seq=0 out_seq=0 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.516+0000 7f3391ffb640 1 -- 172.21.15.78:0/1148997974 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f22b8a6890 con 0x55f22b94f630 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.516+0000 7f3391ffb640 10 client.16053 handle_command_reply: tid=0 2024-09-16T01:16:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.516+0000 7f33b6bcc740 1 client.16053 shutdown 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 msgr2=0x55f22b93c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b93c840 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f33a400ba00 tx=0x7f33a4005230 comp rx=0 tx=0).stop 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 shutdown_connections 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b94f630 0x55f22b951a20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b93fa30 0x55f22b93d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f3391ffb640 0 client.16053 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b93cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/1148997974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b93c840 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 >> 172.21.15.78:0/1148997974 conn(0x55f22b930de0 msgr2=0x55f22b936ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 shutdown_connections 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 1 -- 172.21.15.78:0/1148997974 wait complete. 2024-09-16T01:16:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.517+0000 7f33b6bcc740 20 client.16053 trim_cache size 0 max 16384 2024-09-16T01:16:10.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 Processor -- start 2024-09-16T01:16:10.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 -- start start 2024-09-16T01:16:10.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b934850 0x55f22b92fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b957610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33980128d0 con 0x55f22b93fa30 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f33980101a0 con 0x55f22b935170 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f339800dff0 con 0x55f22b934850 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:55514/0 (socket says 172.21.15.78:55514) 2024-09-16T01:16:10.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b1d8e640 1 -- 172.21.15.78:0/3447257375 learned_addr learned my addr 172.21.15.78:0/3447257375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.541+0000 7f33b2d90640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b957610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 -- 172.21.15.78:0/3447257375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b934850 msgr2=0x55f22b92fdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b258f640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b934850 0x55f22b92fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b934850 0x55f22b92fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 -- 172.21.15.78:0/3447257375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 msgr2=0x55f22b957610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b957610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 -- 172.21.15.78:0/3447257375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22b930c10 con 0x55f22b935170 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b2d90640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b957610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f33a0098590 tx=0x7f33a00bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd61ea111918d62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f3393fff640 1 -- 172.21.15.78:0/3447257375 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a009c130 con 0x55f22b935170 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f3393fff640 1 -- 172.21.15.78:0/3447257375 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a00b4070 con 0x55f22b935170 2024-09-16T01:16:10.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.542+0000 7f3393fff640 1 -- 172.21.15.78:0/3447257375 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a00987a0 con 0x55f22b935170 2024-09-16T01:16:10.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 -- 172.21.15.78:0/3447257375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 msgr2=0x55f22b9570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f33a0098590 tx=0x7f33a00bc040 comp rx=0 tx=0).stop 2024-09-16T01:16:10.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 -- 172.21.15.78:0/3447257375 shutdown_connections 2024-09-16T01:16:10.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b93fa30 0x55f22b957610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b935170 0x55f22b9570d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.543+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/3447257375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b934850 0x55f22b92fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 -- 172.21.15.78:0/3447257375 >> 172.21.15.78:0/3447257375 conn(0x55f22b930de0 msgr2=0x55f22b92f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 -- 172.21.15.78:0/3447257375 shutdown_connections 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 -- 172.21.15.78:0/3447257375 wait complete. 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 -- start start 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 Processor -- start 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b948230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.544+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b6bcc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b93fa30 0x55f22b943aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b258f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b948230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b6bcc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f339800e660 con 0x55f22b934850 2024-09-16T01:16:10.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54574/0 (socket says 172.21.15.78:54574) 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 -- 172.21.15.78:0/412316106 learned_addr learned my addr 172.21.15.78:0/412316106 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b2d90640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b93fa30 0x55f22b943aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f339800fad0 con 0x55f22b93fa30 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f339800ed30 con 0x55f22b935170 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b6bcc740 10 client.0 _ll_register_callbacks cb 0x7f33a400b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 -- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b93fa30 msgr2=0x55f22b943aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b93fa30 0x55f22b943aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 -- 172.21.15.78:0/412316106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 msgr2=0x55f22b948230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b948230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.545+0000 7f33b1d8e640 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f22b92c3f0 con 0x55f22b935170 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b1d8e640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33a000c6a0 tx=0x7f33a0098500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=726230754f2fe00a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:10.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a009c130 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f33a00987a0 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33a00b4070 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33a0005310 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 10 client.25537 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f22b9b9ef0 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a00b4070 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33a009b580 con 0x55f22b935170 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 10 client.25537 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 10 client.25537 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 20 client.25537 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b9ba4d0 0x55f22b9bc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 4 client.25537 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-16T01:16:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.546+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x55f22b8a0be0 con 0x55f22b9ba4d0 2024-09-16T01:16:10.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.547+0000 7f33b258f640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b9ba4d0 0x55f22b9bc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:10.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.547+0000 7f3391ffb640 10 client.25537 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:10.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.547+0000 7f33b258f640 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b9ba4d0 0x55f22b9bc8f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8970b62267caa5ff server_cookie=25bb2e7db0e652c7 in_seq=0 out_seq=0 2024-09-16T01:16:10.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f3391ffb640 1 -- 172.21.15.78:0/412316106 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55f22b8a0be0 con 0x55f22b9ba4d0 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f3391ffb640 10 client.25537 handle_command_reply: tid=0 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 client.25537 shutdown 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 msgr2=0x55f22b943560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33a000c6a0 tx=0x7f33a0098500 comp rx=0 tx=0).stop 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 shutdown_connections 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f22b9ba4d0 0x55f22b9bc8f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f22b93fa30 0x55f22b943aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f22b935170 0x55f22b943560 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 --2- 172.21.15.78:0/412316106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f22b934850 0x55f22b948230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f3391ffb640 0 client.25537 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 >> 172.21.15.78:0/412316106 conn(0x55f22b930de0 msgr2=0x55f22b92fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 shutdown_connections 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 1 -- 172.21.15.78:0/412316106 wait complete. 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.877+0000 7f33b6bcc740 20 client.25537 trim_cache size 0 max 16384 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 22, 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "527487c": { 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 22, 2024-09-16T01:16:10.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 145.3, 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:10.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:10.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:10.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:10.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:10.883 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 msgr2=0x7f33940a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 0x7f33940a5a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f339c006010 tx=0x7f339c0098e0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 msgr2=0x7f33ac073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f33a000e9c0 tx=0x7f33a000ee80 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 shutdown_connections 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f33940a3540 0x7f33940a5a00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33ac106b70 0x7f33ac06cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33ac100a50 0x7f33ac073e50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 --2- 172.21.15.78:0/1747159839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33ac100090 0x7f33ac073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 >> 172.21.15.78:0/1747159839 conn(0x7f33ac077220 msgr2=0x7f33ac1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 shutdown_connections 2024-09-16T01:16:10.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:10.878+0000 7f33b47f3640 1 -- 172.21.15.78:0/1747159839 wait complete. 2024-09-16T01:16:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 295.0 s, 719.6 tps, lat 389.189 ms stddev 2503.207 2024-09-16T01:16:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:11 smithi078 ceph-mon[30937]: pgmap v164: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 891 op/s 2024-09-16T01:16:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:11 smithi193 ceph-mon[35701]: pgmap v164: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 891 op/s 2024-09-16T01:16:11.630 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '527487c', quiesce took 4.9 seconds. Will release after: 20.450000000000003 2024-09-16T01:16:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:11 smithi148 ceph-mon[35030]: pgmap v164: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 13 MiB/s wr, 891 op/s 2024-09-16T01:16:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:13 smithi078 ceph-mon[30937]: pgmap v165: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 673 op/s 2024-09-16T01:16:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:13 smithi193 ceph-mon[35701]: pgmap v165: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 673 op/s 2024-09-16T01:16:13.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:13 smithi148 ceph-mon[35030]: pgmap v165: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.3 MiB/s wr, 673 op/s 2024-09-16T01:16:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:15 smithi078 ceph-mon[30937]: pgmap v166: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.3 MiB/s wr, 634 op/s 2024-09-16T01:16:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:15 smithi193 ceph-mon[35701]: pgmap v166: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.3 MiB/s wr, 634 op/s 2024-09-16T01:16:15.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:15 smithi148 ceph-mon[35030]: pgmap v166: 129 pgs: 129 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.3 MiB/s wr, 634 op/s 2024-09-16T01:16:15.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 300.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:16:17.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:17 smithi078 ceph-mon[30937]: pgmap v167: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 8.3 MiB/s wr, 716 op/s 2024-09-16T01:16:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:17 smithi193 ceph-mon[35701]: pgmap v167: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 8.3 MiB/s wr, 716 op/s 2024-09-16T01:16:17.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:17 smithi148 ceph-mon[35030]: pgmap v167: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 8.3 MiB/s wr, 716 op/s 2024-09-16T01:16:19.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:19 smithi078 ceph-mon[30937]: pgmap v168: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.7 MiB/s wr, 654 op/s 2024-09-16T01:16:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:19 smithi193 ceph-mon[35701]: pgmap v168: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.7 MiB/s wr, 654 op/s 2024-09-16T01:16:19.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:19 smithi148 ceph-mon[35030]: pgmap v168: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 7.7 MiB/s wr, 654 op/s 2024-09-16T01:16:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 305.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:16:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 -- 172.21.15.78:0/3735850697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d81009f0 msgr2=0x7fd8d8100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/3735850697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d81009f0 0x7fd8d8100e50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009930 tx=0x7fd8cc02a420 comp rx=0 tx=0).stop 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 -- 172.21.15.78:0/3735850697 shutdown_connections 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/3735850697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d8101390 0x7fd8d810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/3735850697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d81009f0 0x7fd8d8100e50 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/3735850697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 0x7fd8d8106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 -- 172.21.15.78:0/3735850697 >> 172.21.15.78:0/3735850697 conn(0x7fd8d80fc6b0 msgr2=0x7fd8d80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.390+0000 7fd8dff5f640 1 -- 172.21.15.78:0/3735850697 shutdown_connections 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 -- 172.21.15.78:0/3735850697 wait complete. 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 Processor -- start 2024-09-16T01:16:21.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 -- start start 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d81009f0 0x7fd8d8197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 0x7fd8d819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8110e40 con 0x7fd8d8101390 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8110cc0 con 0x7fd8d8106a00 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dff5f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8110fc0 con 0x7fd8d81009f0 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dd4fa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dd4fa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58118/0 (socket says 172.21.15.78:58118) 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/526444073 learned_addr learned my addr 172.21.15.78:0/526444073 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.391+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d81009f0 0x7fd8d8197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 0x7fd8d819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/526444073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d81009f0 msgr2=0x7fd8d8197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d81009f0 0x7fd8d8197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/526444073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 msgr2=0x7fd8d819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 0x7fd8d819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/526444073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d819c220 con 0x7fd8d8101390 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009230 tx=0x7fd8cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8caffd640 1 -- 172.21.15.78:0/526444073 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc03f030 con 0x7fd8d8101390 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d819c4b0 con 0x7fd8d8101390 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8caffd640 1 -- 172.21.15.78:0/526444073 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8cc003ad0 con 0x7fd8d8101390 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8caffd640 1 -- 172.21.15.78:0/526444073 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc035070 con 0x7fd8d8101390 2024-09-16T01:16:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.392+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d81a3da0 con 0x7fd8d8101390 2024-09-16T01:16:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8caffd640 1 -- 172.21.15.78:0/526444073 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fd8cc042030 con 0x7fd8d8101390 2024-09-16T01:16:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 Processor -- start 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 -- start start 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817baf0 0x5577e817bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817c410 0x5577e8186770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5577e80e02f0 con 0x5577e8186cb0 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5577e8088270 con 0x5577e817baf0 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577e80dc2b0 con 0x5577e817c410 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8dd4fa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8dd4fa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58130/0 (socket says 172.21.15.78:58130) 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8de4fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817baf0 0x5577e817bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.395+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/1828842182 learned_addr learned my addr 172.21.15.78:0/1828842182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8caffd640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 0x7fd8c40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817c410 0x5577e8186770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 0x7fd8c40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/1828842182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817c410 msgr2=0x5577e8186770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817c410 0x5577e8186770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/1828842182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817baf0 msgr2=0x5577e817bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817baf0 0x5577e817bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 -- 172.21.15.78:0/1828842182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577e8087340 con 0x5577e8186cb0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8caffd640 1 -- 172.21.15.78:0/526444073 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd8cc0f0450 con 0x7fd8d8101390 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0b88a0 tx=0x7fd8cc07c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b84aec0d9c9b1db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 0x7fd8c40a5920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00098b0 tx=0x7fd8d0004ab0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8c3fff640 1 -- 172.21.15.78:0/1828842182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc049020 con 0x5577e8186cb0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8c3fff640 1 -- 172.21.15.78:0/1828842182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8cc03b460 con 0x5577e8186cb0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.396+0000 7fd8c3fff640 1 -- 172.21.15.78:0/1828842182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc034170 con 0x5577e8186cb0 2024-09-16T01:16:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1828842182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 msgr2=0x5577e81890a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0b88a0 tx=0x7fd8cc07c2d0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1828842182 shutdown_connections 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e81890a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817c410 0x5577e8186770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1828842182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817baf0 0x5577e817bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1828842182 >> 172.21.15.78:0/1828842182 conn(0x5577e8178060 msgr2=0x5577e8178440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1828842182 shutdown_connections 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1828842182 wait complete. 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.397+0000 7fd8e2dd6740 1 -- start start 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 Processor -- start 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e8184060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e81845a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5577e8189640 con 0x5577e817baf0 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5577e8069290 con 0x5577e817c410 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577e81897c0 con 0x5577e8186cb0 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8e2dd6740 10 client.0 _ll_register_callbacks cb 0x5577e809a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58134/0 (socket says 172.21.15.78:58134) 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/898447620 learned_addr learned my addr 172.21.15.78:0/898447620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e8184060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/898447620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 msgr2=0x5577e81845a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e81845a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 msgr2=0x5577e8184060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e8184060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/898447620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577e7df3210 con 0x5577e817baf0 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.398+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e8184060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4010a80 tx=0x7fd8d4010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef05170bb2475322 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d401e060 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd8d400fc50 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5577e80eda20 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd8d4025050 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 10 client.16071 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5577e7ebf620 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5577e80eda20 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd8d400e420 con 0x5577e817baf0 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 10 client.16071 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 20 client.16071 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e8196920 0x5577e8198d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 4 client.16071 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.399+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577e80e7d70 con 0x5577e8196920 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.400+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e8196920 0x5577e8198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.400+0000 7fd8c17fa640 10 client.16071 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:21.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.400+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e8196920 0x5577e8198d10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccde5751fef79656 server_cookie=4b9aa22d7bb17dcd in_seq=0 out_seq=0 2024-09-16T01:16:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8c17fa640 1 -- 172.21.15.78:0/898447620 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5577e80e7d70 con 0x5577e8196920 2024-09-16T01:16:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8c17fa640 10 client.16071 handle_command_reply: tid=0 2024-09-16T01:16:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 client.16071 shutdown 2024-09-16T01:16:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 msgr2=0x5577e8183b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4010a80 tx=0x7fd8d4010f40 comp rx=0 tx=0).stop 2024-09-16T01:16:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 shutdown_connections 2024-09-16T01:16:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e8196920 0x5577e8198d10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e81845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e8184060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8c17fa640 0 client.16071 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/898447620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e8183b20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 >> 172.21.15.78:0/898447620 conn(0x5577e8178060 msgr2=0x5577e817ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 shutdown_connections 2024-09-16T01:16:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/898447620 wait complete. 2024-09-16T01:16:21.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.401+0000 7fd8e2dd6740 20 client.16071 trim_cache size 0 max 16384 2024-09-16T01:16:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 Processor -- start 2024-09-16T01:16:21.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 -- start start 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817baf0 0x5577e81843d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e817dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81b06d0 con 0x5577e8186cb0 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81adfa0 con 0x5577e817c410 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81abdf0 con 0x5577e817baf0 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58138/0 (socket says 172.21.15.78:58138) 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 -- 172.21.15.78:0/4024921525 learned_addr learned my addr 172.21.15.78:0/4024921525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e817dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817baf0 0x5577e81843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 -- 172.21.15.78:0/4024921525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817baf0 msgr2=0x5577e81843d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817baf0 0x5577e81843d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 -- 172.21.15.78:0/4024921525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 msgr2=0x5577e817dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e817dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 -- 172.21.15.78:0/4024921525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577e8177e90 con 0x5577e8186cb0 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.416+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0003c00 tx=0x7fd8d0009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd69acb153ee95e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8c37fe640 1 -- 172.21.15.78:0/4024921525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d001a040 con 0x5577e8186cb0 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8c37fe640 1 -- 172.21.15.78:0/4024921525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8d0004030 con 0x5577e8186cb0 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8c37fe640 1 -- 172.21.15.78:0/4024921525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d0005280 con 0x5577e8186cb0 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/4024921525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 msgr2=0x5577e818ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0003c00 tx=0x7fd8d0009600 comp rx=0 tx=0).stop 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/4024921525 shutdown_connections 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e8186cb0 0x5577e818ec00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e817dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/4024921525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e817baf0 0x5577e81843d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/4024921525 >> 172.21.15.78:0/4024921525 conn(0x5577e8178060 msgr2=0x5577e817d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/4024921525 shutdown_connections 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/4024921525 wait complete. 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.417+0000 7fd8e2dd6740 1 -- start start 2024-09-16T01:16:21.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 Processor -- start 2024-09-16T01:16:21.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e818f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e818bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81ac460 con 0x5577e817baf0 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81ad8d0 con 0x5577e817c410 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81acb30 con 0x5577e8186cb0 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8e2dd6740 10 client.0 _ll_register_callbacks cb 0x7fd8d400a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58154/0 (socket says 172.21.15.78:58154) 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/1749309225 learned_addr learned my addr 172.21.15.78:0/1749309225 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e818f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8de4fc640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e818bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/1749309225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 msgr2=0x5577e818bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e818bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 msgr2=0x5577e818f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e818f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 -- 172.21.15.78:0/1749309225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577e81791f0 con 0x5577e817baf0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8ddcfb640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400f8e0 tx=0x7fd8d40106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5a691415eb605b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.418+0000 7fd8c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d402d020 con 0x5577e817baf0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fd8d400ec20 con 0x5577e817baf0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8d0004030 con 0x5577e817baf0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd8d401e020 con 0x5577e817baf0 2024-09-16T01:16:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 10 client.16077 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5577e8203f40 con 0x5577e817baf0 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8d0004030 con 0x5577e817baf0 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd8d401a9c0 con 0x5577e817baf0 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 10 client.16077 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 20 client.16077 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e82046e0 0x5577e8206b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 4 client.16077 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577e8064900 con 0x5577e82046e0 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.419+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e82046e0 0x5577e8206b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.420+0000 7fd8c17fa640 10 client.16077 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.420+0000 7fd8dd4fa640 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e82046e0 0x5577e8206b00 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc76da1bcbfa2e9a server_cookie=34205b121ff541f8 in_seq=0 out_seq=0 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.427+0000 7fd8c17fa640 1 -- 172.21.15.78:0/1749309225 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577e8064900 con 0x5577e82046e0 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.427+0000 7fd8c17fa640 10 client.16077 handle_command_reply: tid=0 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.427+0000 7fd8e2dd6740 1 client.16077 shutdown 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.427+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 msgr2=0x5577e818ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.427+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd8d400f8e0 tx=0x7fd8d40106b0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 shutdown_connections 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5577e82046e0 0x5577e8206b00 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577e8186cb0 0x5577e818bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5577e817c410 0x5577e818f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 --2- 172.21.15.78:0/1749309225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e817baf0 0x5577e818ec10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8c17fa640 0 client.16077 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 >> 172.21.15.78:0/1749309225 conn(0x5577e8178060 msgr2=0x5577e81771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 shutdown_connections 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 1 -- 172.21.15.78:0/1749309225 wait complete. 2024-09-16T01:16:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8e2dd6740 20 client.16077 trim_cache size 0 max 16384 2024-09-16T01:16:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:16:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:16:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:16:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 msgr2=0x7fd8c40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 0x7fd8c40a5920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00098b0 tx=0x7fd8d0004ab0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 msgr2=0x7fd8d8197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009230 tx=0x7fd8cc004270 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 shutdown_connections 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd8c40a3460 0x7fd8c40a5920 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8d8106a00 0x7fd8d819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8d8101390 0x7fd8d8197770 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 --2- 172.21.15.78:0/526444073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d81009f0 0x7fd8d8197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 >> 172.21.15.78:0/526444073 conn(0x7fd8d80fc6b0 msgr2=0x7fd8d80fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 shutdown_connections 2024-09-16T01:16:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:21.428+0000 7fd8dff5f640 1 -- 172.21.15.78:0/526444073 wait complete. 2024-09-16T01:16:21.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:21 smithi078 ceph-mon[30937]: pgmap v169: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.7 MiB/s wr, 658 op/s 2024-09-16T01:16:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:21 smithi193 ceph-mon[35701]: pgmap v169: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.7 MiB/s wr, 658 op/s 2024-09-16T01:16:21.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:21 smithi148 ceph-mon[35030]: pgmap v169: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 7.7 MiB/s wr, 658 op/s 2024-09-16T01:16:23.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:23 smithi078 ceph-mon[30937]: pgmap v170: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-09-16T01:16:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:23 smithi193 ceph-mon[35701]: pgmap v170: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-09-16T01:16:23.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:23 smithi148 ceph-mon[35030]: pgmap v170: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 1.9 MiB/s wr, 167 op/s 2024-09-16T01:16:25.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:25 smithi078 ceph-mon[30937]: pgmap v171: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-09-16T01:16:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:25 smithi193 ceph-mon[35701]: pgmap v171: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-09-16T01:16:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:25 smithi148 ceph-mon[35030]: pgmap v171: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 1.0 MiB/s wr, 102 op/s 2024-09-16T01:16:25.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 310.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:16:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:27 smithi078 ceph-mon[30937]: pgmap v172: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-09-16T01:16:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:27 smithi193 ceph-mon[35701]: pgmap v172: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-09-16T01:16:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:27 smithi148 ceph-mon[35030]: pgmap v172: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-09-16T01:16:29.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:29 smithi078 ceph-mon[30937]: pgmap v173: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 626 KiB/s wr, 76 op/s 2024-09-16T01:16:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:29 smithi193 ceph-mon[35701]: pgmap v173: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 626 KiB/s wr, 76 op/s 2024-09-16T01:16:29.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:29 smithi148 ceph-mon[35030]: pgmap v173: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 626 KiB/s wr, 76 op/s 2024-09-16T01:16:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 315.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:16:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:31 smithi078 ceph-mon[30937]: pgmap v174: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 631 KiB/s wr, 79 op/s 2024-09-16T01:16:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:31 smithi193 ceph-mon[35701]: pgmap v174: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 631 KiB/s wr, 79 op/s 2024-09-16T01:16:31.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:31 smithi148 ceph-mon[35030]: pgmap v174: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 631 KiB/s wr, 79 op/s 2024-09-16T01:16:32.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:16:32.346 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- 172.21.15.78:0/2842153784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 msgr2=0x7f366c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 --2- 172.21.15.78:0/2842153784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c10da40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3664009ca0 tx=0x7f366402a950 comp rx=0 tx=0).stop 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- 172.21.15.78:0/2842153784 shutdown_connections 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 --2- 172.21.15.78:0/2842153784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c10da40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 --2- 172.21.15.78:0/2842153784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 0x7f366c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 --2- 172.21.15.78:0/2842153784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 0x7f366c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- 172.21.15.78:0/2842153784 >> 172.21.15.78:0/2842153784 conn(0x7f366c0fc7d0 msgr2=0x7f366c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- 172.21.15.78:0/2842153784 shutdown_connections 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- 172.21.15.78:0/2842153784 wait complete. 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 Processor -- start 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.835+0000 7f36745f3640 1 -- start start 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 0x7f366c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 0x7f366c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f366c110fa0 con 0x7f366c102ef0 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f366c110e20 con 0x7f366c100e20 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36745f3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f366c111120 con 0x7f366c102570 2024-09-16T01:16:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48752/0 (socket says 172.21.15.78:48752) 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 -- 172.21.15.78:0/1283052155 learned_addr learned my addr 172.21.15.78:0/1283052155 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f367238f640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 0x7f366c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3671b8e640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 0x7f366c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 -- 172.21.15.78:0/1283052155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 msgr2=0x7f366c19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 0x7f366c19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 -- 172.21.15.78:0/1283052155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 msgr2=0x7f366c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 0x7f366c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 -- 172.21.15.78:0/1283052155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c19f680 con 0x7f366c102ef0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f3672b90640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3664006010 tx=0x7f366402f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.836+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664034040 con 0x7f366c102ef0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.837+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366c19f910 con 0x7f366c102ef0 2024-09-16T01:16:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.837+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3664003e00 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.837+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664045480 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.837+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366c1a7250 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.838+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366c101200 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.839+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3664037020 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.839+0000 7f36637fe640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 0x7f36580a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.840+0000 7f367238f640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 0x7f36580a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.840+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f36640f1110 con 0x7f366c102ef0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.840+0000 7f367238f640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 0x7f36580a5870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36680058c0 tx=0x7f366800a2d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:32.843+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f36640bd900 con 0x7f366c102ef0 2024-09-16T01:16:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.043+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f366c19ba30 con 0x7f366c102ef0 2024-09-16T01:16:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.044+0000 7f36637fe640 1 -- 172.21.15.78:0/1283052155 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f366c19ba30 con 0x7f366c102ef0 2024-09-16T01:16:33.046 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:16:33.046 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:16:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 msgr2=0x7f36580a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 0x7f36580a5870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f36680058c0 tx=0x7f366800a2d0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 msgr2=0x7f366c19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3664006010 tx=0x7f366402f6f0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 shutdown_connections 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f36580a33b0 0x7f36580a5870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f366c102ef0 0x7f366c19efa0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f366c102570 0x7f366c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 --2- 172.21.15.78:0/1283052155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f366c100e20 0x7f366c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.046+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 >> 172.21.15.78:0/1283052155 conn(0x7f366c0fc7d0 msgr2=0x7f366c06a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.047+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 shutdown_connections 2024-09-16T01:16:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:33.047+0000 7f36745f3640 1 -- 172.21.15.78:0/1283052155 wait complete. 2024-09-16T01:16:33.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:33 smithi078 ceph-mon[30937]: pgmap v175: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 629 KiB/s wr, 75 op/s 2024-09-16T01:16:33.491 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:33 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1283052155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:33 smithi193 ceph-mon[35701]: pgmap v175: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 629 KiB/s wr, 75 op/s 2024-09-16T01:16:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:33 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1283052155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:33.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:33 smithi148 ceph-mon[35030]: pgmap v175: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 629 KiB/s wr, 75 op/s 2024-09-16T01:16:33.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:33 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1283052155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:16:33.894 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 527487c --release --await' 2024-09-16T01:16:33.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 527487c --release --await 2024-09-16T01:16:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- 172.21.15.78:0/1671888824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89341080f0 msgr2=0x7f893410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- 172.21.15.78:0/1671888824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89341080f0 0x7f893410ea40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f892c009cd0 tx=0x7f892c02a970 comp rx=0 tx=0).stop 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- 172.21.15.78:0/1671888824 shutdown_connections 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- 172.21.15.78:0/1671888824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89341080f0 0x7f893410ea40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- 172.21.15.78:0/1671888824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8934101150 0x7f8934107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- 172.21.15.78:0/1671888824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 0x7f8934100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- 172.21.15.78:0/1671888824 >> 172.21.15.78:0/1671888824 conn(0x7f8934075f20 msgr2=0x7f8934076330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- 172.21.15.78:0/1671888824 shutdown_connections 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- 172.21.15.78:0/1671888824 wait complete. 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 Processor -- start 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- start start 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 0x7f8934198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 0x7f893419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8934111f20 con 0x7f8934101150 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8934111da0 con 0x7f89341080f0 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.515+0000 7f893a5af640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89341120a0 con 0x7f8934100830 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48960/0 (socket says 172.21.15.78:48960) 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 -- 172.21.15.78:0/2049388078 learned_addr learned my addr 172.21.15.78:0/2049388078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8933fff640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 0x7f8934198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8938b4c640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 0x7f893419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 -- 172.21.15.78:0/2049388078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 msgr2=0x7f8934198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 0x7f8934198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 -- 172.21.15.78:0/2049388078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 msgr2=0x7f893419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 0x7f893419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 -- 172.21.15.78:0/2049388078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893419d300 con 0x7f8934101150 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8938b4c640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 0x7f893419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f892bfff640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f891c00c970 tx=0x7f891c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8931ffb640 1 -- 172.21.15.78:0/2049388078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c014070 con 0x7f8934101150 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8931ffb640 1 -- 172.21.15.78:0/2049388078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f891c004440 con 0x7f8934101150 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893419d5f0 con 0x7f8934101150 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f8931ffb640 1 -- 172.21.15.78:0/2049388078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c00f450 con 0x7f8934101150 2024-09-16T01:16:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.516+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89341a4e80 con 0x7f8934101150 2024-09-16T01:16:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.518+0000 7f8931ffb640 1 -- 172.21.15.78:0/2049388078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f891c017070 con 0x7f8934101150 2024-09-16T01:16:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 Processor -- start 2024-09-16T01:16:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 -- start start 2024-09-16T01:16:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 0x55e00b5e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e00b5462f0 con 0x55e00b5e22d0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e00b4ee270 con 0x55e00b5e19b0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f893d424740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e00b5422b0 con 0x55e00b5f10c0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8931ffb640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 0x7f89200a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48970/0 (socket says 172.21.15.78:48970) 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 -- 172.21.15.78:0/798209644 learned_addr learned my addr 172.21.15.78:0/798209644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8938b4c640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 0x55e00b5e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8938b4c640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 0x7f89200a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8931ffb640 1 -- 172.21.15.78:0/2049388078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f891c0cff30 con 0x7f8934101150 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f892bfff640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 -- 172.21.15.78:0/798209644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 msgr2=0x55e00b5f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 -- 172.21.15.78:0/798209644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 msgr2=0x55e00b5e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 0x55e00b5e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 -- 172.21.15.78:0/798209644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e00b4ed340 con 0x55e00b5e22d0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8938b4c640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 0x55e00b5e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.519+0000 7f8933fff640 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f892400cbe0 tx=0x7f892400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd0047e04712c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f8938b4c640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 0x7f89200a5920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f892c02e4d0 tx=0x7f892c02a970 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f892a7fc640 1 -- 172.21.15.78:0/798209644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924010630 con 0x55e00b5e22d0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f892a7fc640 1 -- 172.21.15.78:0/798209644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924010cb0 con 0x55e00b5e22d0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f892a7fc640 1 -- 172.21.15.78:0/798209644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892401ac20 con 0x55e00b5e22d0 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 -- 172.21.15.78:0/798209644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 msgr2=0x55e00b5f0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f892400cbe0 tx=0x7f892400e720 comp rx=0 tx=0).stop 2024-09-16T01:16:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 -- 172.21.15.78:0/798209644 shutdown_connections 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e22d0 0x55e00b5f0b80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 --2- 172.21.15.78:0/798209644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e19b0 0x55e00b5e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 -- 172.21.15.78:0/798209644 >> 172.21.15.78:0/798209644 conn(0x55e00b5de060 msgr2=0x55e00b5de440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 -- 172.21.15.78:0/798209644 shutdown_connections 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.520+0000 7f893d424740 1 -- 172.21.15.78:0/798209644 wait complete. 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 -- start start 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 Processor -- start 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5eea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e00b5f3c90 con 0x55e00b5e19b0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e00b4cf290 con 0x55e00b5f10c0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e00b5f3e10 con 0x55e00b5e22d0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f893d424740 10 client.0 _ll_register_callbacks cb 0x55e00b500140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48986/0 (socket says 172.21.15.78:48986) 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 -- 172.21.15.78:0/3651089082 learned_addr learned my addr 172.21.15.78:0/3651089082 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 -- 172.21.15.78:0/3651089082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 msgr2=0x55e00b5ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8933fff640 1 -- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 msgr2=0x55e00b5eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.521+0000 7f8938b4c640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f8933fff640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f8933fff640 1 -- 172.21.15.78:0/3651089082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e00b259210 con 0x55e00b5e19b0 2024-09-16T01:16:34.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f8938b4c640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f8933fff640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f892400c920 tx=0x7f8924011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aedb6456cfda475 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892401ab20 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f892401acc0 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8924010cb0 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8924030050 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 10 client.16095 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e00b325620 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924010cb0 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8924025890 con 0x55e00b5e19b0 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 10 client.16095 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 10 client.16095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 20 client.16095 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:34.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b600f10 0x55e00b603300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 4 client.16095 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.522+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e00b553a20 con 0x55e00b600f10 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.523+0000 7f892bfff640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b600f10 0x55e00b603300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.523+0000 7f892bfff640 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b600f10 0x55e00b603300 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2755982eab9e7496 server_cookie=aff14420d19d5634 in_seq=0 out_seq=0 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.523+0000 7f89137fe640 10 client.16095 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f89137fe640 1 -- 172.21.15.78:0/3651089082 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e00b553a20 con 0x55e00b600f10 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f89137fe640 10 client.16095 handle_command_reply: tid=0 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 client.16095 shutdown 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 msgr2=0x55e00b5ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f892400c920 tx=0x7f8924011e70 comp rx=0 tx=0).stop 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 shutdown_connections 2024-09-16T01:16:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b600f10 0x55e00b603300 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f89137fe640 0 client.16095 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 --2- 172.21.15.78:0/3651089082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5ee000 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 >> 172.21.15.78:0/3651089082 conn(0x55e00b5de060 msgr2=0x55e00b5e3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 shutdown_connections 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.524+0000 7f893d424740 1 -- 172.21.15.78:0/3651089082 wait complete. 2024-09-16T01:16:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.525+0000 7f893d424740 20 client.16095 trim_cache size 0 max 16384 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 Processor -- start 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 -- start start 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e22d0 0x55e00b608e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5dc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f89341af560 con 0x55e00b5e19b0 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f89341ace30 con 0x55e00b5e22d0 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.540+0000 7f893d424740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89341aac80 con 0x55e00b5f10c0 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48990/0 (socket says 172.21.15.78:48990) 2024-09-16T01:16:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 -- 172.21.15.78:0/1837643738 learned_addr learned my addr 172.21.15.78:0/1837643738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 -- 172.21.15.78:0/1837643738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 msgr2=0x55e00b5dc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5dc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f892bfff640 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e22d0 0x55e00b608e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 -- 172.21.15.78:0/1837643738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e22d0 msgr2=0x55e00b608e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e22d0 0x55e00b608e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 -- 172.21.15.78:0/1837643738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e00b5e2100 con 0x55e00b5e19b0 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8933fff640 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f892401a8d0 tx=0x7f892400c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28529441e9f7425 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8929ffb640 1 -- 172.21.15.78:0/1837643738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892400f3c0 con 0x55e00b5e19b0 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8929ffb640 1 -- 172.21.15.78:0/1837643738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924004020 con 0x55e00b5e19b0 2024-09-16T01:16:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.541+0000 7f8929ffb640 1 -- 172.21.15.78:0/1837643738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924005040 con 0x55e00b5e19b0 2024-09-16T01:16:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 -- 172.21.15.78:0/1837643738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 msgr2=0x55e00b608900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f892401a8d0 tx=0x7f892400c4d0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 -- 172.21.15.78:0/1837643738 shutdown_connections 2024-09-16T01:16:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5f10c0 0x55e00b5dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5e22d0 0x55e00b608e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 --2- 172.21.15.78:0/1837643738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b608900 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 -- 172.21.15.78:0/1837643738 >> 172.21.15.78:0/1837643738 conn(0x55e00b5de060 msgr2=0x55e00b5e9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 -- 172.21.15.78:0/1837643738 shutdown_connections 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.542+0000 7f893d424740 1 -- 172.21.15.78:0/1837643738 wait complete. 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 -- start start 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 Processor -- start 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5f4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f89341ab2f0 con 0x55e00b5e19b0 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f89341ac760 con 0x55e00b5f10c0 2024-09-16T01:16:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89341ab9c0 con 0x55e00b5e22d0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f893d424740 10 client.0 _ll_register_callbacks cb 0x7f8924014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.543+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49006/0 (socket says 172.21.15.78:49006) 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 -- 172.21.15.78:0/764028692 learned_addr learned my addr 172.21.15.78:0/764028692 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8938b4c640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f892bfff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 -- 172.21.15.78:0/764028692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 msgr2=0x55e00b5f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 -- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 msgr2=0x55e00b5f4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5f4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 -- 172.21.15.78:0/764028692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e00b5f52d0 con 0x55e00b5e19b0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f8933fff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89240058c0 tx=0x7f892400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90831ada5ca5f270 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f89137fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924041060 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f892401ab20 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8924004020 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8924048050 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f893d424740 10 client.16101 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.544+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e00b66b5a0 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8924004020 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8924004660 con 0x55e00b5e19b0 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 10 client.16101 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 10 client.16101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 20 client.16101 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b66bc60 0x55e00b66e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 4 client.16101 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:16:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e00b54dd70 con 0x55e00b66bc60 2024-09-16T01:16:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.545+0000 7f892bfff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b66bc60 0x55e00b66e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.546+0000 7f89137fe640 10 client.16101 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.546+0000 7f892bfff640 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b66bc60 0x55e00b66e080 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=917b0de005ae0385 server_cookie=d8b00af8b88405bc in_seq=0 out_seq=0 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f89137fe640 1 -- 172.21.15.78:0/764028692 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e00b54dd70 con 0x55e00b66bc60 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f89137fe640 10 client.16101 handle_command_reply: tid=0 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 client.16101 shutdown 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 msgr2=0x55e00b5f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89240058c0 tx=0x7f892400f3c0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 shutdown_connections 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e00b66bc60 0x55e00b66e080 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e00b5f10c0 0x55e00b5f4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e00b5e22d0 0x55e00b5f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 --2- 172.21.15.78:0/764028692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e00b5e19b0 0x55e00b5f9530 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 >> 172.21.15.78:0/764028692 conn(0x55e00b5de060 msgr2=0x55e00b5ef860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f89137fe640 0 client.16101 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 shutdown_connections 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.553+0000 7f893d424740 1 -- 172.21.15.78:0/764028692 wait complete. 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893d424740 20 client.16101 trim_cache size 0 max 16384 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:16:34.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 msgr2=0x7f89200a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 0x7f89200a5920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f892c02e4d0 tx=0x7f892c02a970 comp rx=0 tx=0).stop 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 msgr2=0x7f8934198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f891c00c970 tx=0x7f891c00ce30 comp rx=0 tx=0).stop 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 shutdown_connections 2024-09-16T01:16:34.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f89200a3460 0x7f89200a5920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341080f0 0x7f893419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8934101150 0x7f8934198850 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 --2- 172.21.15.78:0/2049388078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8934100830 0x7f8934198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 >> 172.21.15.78:0/2049388078 conn(0x7f8934075f20 msgr2=0x7f89340ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 shutdown_connections 2024-09-16T01:16:34.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:34.554+0000 7f893a5af640 1 -- 172.21.15.78:0/2049388078 wait complete. 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 -- 172.21.15.78:0/3304420276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 msgr2=0x7fb2dc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 --2- 172.21.15.78:0/3304420276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc106e60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009930 tx=0x7fb2d802a420 comp rx=0 tx=0).stop 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 -- 172.21.15.78:0/3304420276 shutdown_connections 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 --2- 172.21.15.78:0/3304420276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 --2- 172.21.15.78:0/3304420276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 0x7fb2dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 --2- 172.21.15.78:0/3304420276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc106e60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 -- 172.21.15.78:0/3304420276 >> 172.21.15.78:0/3304420276 conn(0x7fb2dc0fc7a0 msgr2=0x7fb2dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 -- 172.21.15.78:0/3304420276 shutdown_connections 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.621+0000 7fb2e4482640 1 -- 172.21.15.78:0/3304420276 wait complete. 2024-09-16T01:16:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 Processor -- start 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 -- start start 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 0x7fb2dc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc111fa0 con 0x7fb2dc106a80 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc111e20 con 0x7fb2dc101420 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e4482640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc112120 con 0x7fb2dc100a80 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e2a1f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e2a1f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49028/0 (socket says 172.21.15.78:49028) 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e2a1f640 1 -- 172.21.15.78:0/2154188262 learned_addr learned my addr 172.21.15.78:0/2154188262 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.622+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e221e640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 0x7fb2dc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 -- 172.21.15.78:0/2154188262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 msgr2=0x7fb2dc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 0x7fb2dc198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 -- 172.21.15.78:0/2154188262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 msgr2=0x7fb2dc198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 -- 172.21.15.78:0/2154188262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc19d4f0 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e2a1f640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400cad0 tx=0x7fb2d400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2d37fe640 1 -- 172.21.15.78:0/2154188262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400ae20 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc19d7e0 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2d37fe640 1 -- 172.21.15.78:0/2154188262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d4005020 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2d37fe640 1 -- 172.21.15.78:0/2154188262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400f6f0 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.623+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc1a50c0 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.625+0000 7fb2d37fe640 1 -- 172.21.15.78:0/2154188262 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb2d40051d0 con 0x7fb2dc106a80 2024-09-16T01:16:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 Processor -- start 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 -- start start 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d744440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56191d6a9620 con 0x56191d74f240 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56191d650e50 con 0x56191d744980 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e685b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56191d6a55e0 con 0x56191d744060 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2d37fe640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 0x7fb2c80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49038/0 (socket says 172.21.15.78:49038) 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/2185522804 learned_addr learned my addr 172.21.15.78:0/2185522804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e2a1f640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 0x7fb2c80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2d37fe640 1 -- 172.21.15.78:0/2154188262 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb2d40d0200 con 0x7fb2dc106a80 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d744440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.626+0000 7fb2e221e640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/2185522804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 msgr2=0x56191d744440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d744440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/2185522804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 msgr2=0x56191d74ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/2185522804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56191d641d40 con 0x56191d74f240 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e221e640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e2a1f640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 0x7fb2c80a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb2d409be50 tx=0x7fb2d40bc040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00adf0 tx=0x7fb2cc00a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcf3c56c9c8a870 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2d0ff9640 1 -- 172.21.15.78:0/2185522804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc01c070 con 0x56191d74f240 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2d0ff9640 1 -- 172.21.15.78:0/2185522804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc00fdf0 con 0x56191d74f240 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.627+0000 7fb2d0ff9640 1 -- 172.21.15.78:0/2185522804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc014710 con 0x56191d74f240 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- 172.21.15.78:0/2185522804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 msgr2=0x56191d751630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00adf0 tx=0x7fb2cc00a520 comp rx=0 tx=0).stop 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- 172.21.15.78:0/2185522804 shutdown_connections 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d751630 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 --2- 172.21.15.78:0/2185522804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d744440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- 172.21.15.78:0/2185522804 >> 172.21.15.78:0/2185522804 conn(0x56191d7405f0 msgr2=0x56191d7409d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- 172.21.15.78:0/2185522804 shutdown_connections 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- 172.21.15.78:0/2185522804 wait complete. 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 -- start start 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.628+0000 7fb2e685b740 1 Processor -- start 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d74c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56191d751e10 con 0x56191d74f240 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56191d631f10 con 0x56191d744980 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56191d751f90 con 0x56191d744060 2024-09-16T01:16:34.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 10 client.0 _ll_register_callbacks cb 0x56191d655db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49042/0 (socket says 172.21.15.78:49042) 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/1986334839 learned_addr learned my addr 172.21.15.78:0/1986334839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d74c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/1986334839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 msgr2=0x56191d74c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d74c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 msgr2=0x56191d74c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/1986334839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56191d3b8580 con 0x56191d74f240 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc01a040 tx=0x7fb2cc006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d761ddefd6df64cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:34.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc01c070 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb2cc014dc0 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.629+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2cc00fdf0 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb2cc017460 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 10 client.16113 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56191d482960 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc00fdf0 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb2cc022070 con 0x56191d74f240 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 10 client.16113 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 10 client.16113 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 20 client.16113 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d75ece0 0x56191d7610d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 4 client.16113 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:34.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56191d6b6b90 con 0x56191d75ece0 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.630+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d75ece0 0x56191d7610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2c5ffb640 10 client.16113 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d75ece0 0x56191d7610d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab21a11bad2c612d server_cookie=73e8f1d5b5182900 in_seq=0 out_seq=0 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1986334839 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56191d6b6b90 con 0x56191d75ece0 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2c5ffb640 10 client.16113 handle_command_reply: tid=0 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 client.16113 shutdown 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 msgr2=0x56191d74cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc01a040 tx=0x7fb2cc006f90 comp rx=0 tx=0).stop 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 shutdown_connections 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d75ece0 0x56191d7610d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d74cc90 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d74c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2c5ffb640 0 client.16113 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1986334839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d74c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.631+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 >> 172.21.15.78:0/1986334839 conn(0x56191d7405f0 msgr2=0x56191d7462d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.632+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 shutdown_connections 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.632+0000 7fb2e685b740 1 -- 172.21.15.78:0/1986334839 wait complete. 2024-09-16T01:16:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.632+0000 7fb2e685b740 20 client.16113 trim_cache size 0 max 16384 2024-09-16T01:16:34.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 Processor -- start 2024-09-16T01:16:34.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 -- start start 2024-09-16T01:16:34.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d7575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d757b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1b19f0 con 0x56191d74f240 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1af2c0 con 0x56191d744980 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e685b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1ad110 con 0x56191d744060 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.648+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49058/0 (socket says 172.21.15.78:49058) 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/3598612694 learned_addr learned my addr 172.21.15.78:0/3598612694 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d7575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e221e640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d757b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/3598612694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 msgr2=0x56191d7575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d7575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/3598612694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 msgr2=0x56191d757b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d757b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 -- 172.21.15.78:0/3598612694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56191d743e90 con 0x56191d74f240 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e221e640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d757b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0144c0 tx=0x7fb2cc0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5efaad19bc4fe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2c7fff640 1 -- 172.21.15.78:0/3598612694 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc01c070 con 0x56191d74f240 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2c7fff640 1 -- 172.21.15.78:0/3598612694 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc01d040 con 0x56191d74f240 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.649+0000 7fb2c7fff640 1 -- 172.21.15.78:0/3598612694 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00fe40 con 0x56191d74f240 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- 172.21.15.78:0/3598612694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 msgr2=0x56191d7667f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0144c0 tx=0x7fb2cc0047d0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- 172.21.15.78:0/3598612694 shutdown_connections 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d74f240 0x56191d7667f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d744980 0x56191d757b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 --2- 172.21.15.78:0/3598612694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d7575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- 172.21.15.78:0/3598612694 >> 172.21.15.78:0/3598612694 conn(0x56191d7405f0 msgr2=0x56191d74c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- 172.21.15.78:0/3598612694 shutdown_connections 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- 172.21.15.78:0/3598612694 wait complete. 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 -- start start 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.650+0000 7fb2e685b740 1 Processor -- start 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d749b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 0x56191d7489c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1ad780 con 0x56191d744980 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1aebf0 con 0x56191d74f240 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1ade50 con 0x56191d744060 2024-09-16T01:16:34.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 10 client.0 _ll_register_callbacks cb 0x7fb2cc00f0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49062/0 (socket says 172.21.15.78:49062) 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 -- 172.21.15.78:0/1800000510 learned_addr learned my addr 172.21.15.78:0/1800000510 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 0x56191d7489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 -- 172.21.15.78:0/1800000510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 msgr2=0x56191d749b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d749b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 -- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 msgr2=0x56191d7489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 0x56191d7489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 -- 172.21.15.78:0/1800000510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56191d749000 con 0x56191d744980 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 0x56191d7489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e221e640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8006010 tx=0x7fb2d8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc448114f9dee3f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d804b060 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2cc01d040 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7fb2d8005040 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb2d8052050 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.651+0000 7fb2e685b740 10 client.16119 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56191d7ce6d0 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc01d040 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb2d8045070 con 0x56191d744980 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 10 client.16119 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 10 client.16119 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 20 client.16119 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d7cedd0 0x56191d7d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 4 client.16119 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "527487c", "release": true, "await": true}] 2024-09-16T01:16:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "527487c", "release": true, "await": true}) -- 0x56191d6b10a0 con 0x56191d7cedd0 2024-09-16T01:16:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.652+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d7cedd0 0x56191d7d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.653+0000 7fb2e1a1d640 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d7cedd0 0x56191d7d11f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663b068fc16d0516 server_cookie=ec4aff6fae67a93e in_seq=0 out_seq=0 2024-09-16T01:16:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.653+0000 7fb2c5ffb640 10 client.16119 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.706+0000 7fb2c5ffb640 1 -- 172.21.15.78:0/1800000510 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56191d6b10a0 con 0x56191d7cedd0 2024-09-16T01:16:34.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.706+0000 7fb2c5ffb640 10 client.16119 handle_command_reply: tid=0 2024-09-16T01:16:34.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.706+0000 7fb2e685b740 1 client.16119 shutdown 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 msgr2=0x56191d748480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8006010 tx=0x7fb2d8004040 comp rx=0 tx=0).stop 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 shutdown_connections 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56191d7cedd0 0x56191d7d11f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56191d74f240 0x56191d7489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56191d744980 0x56191d748480 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 --2- 172.21.15.78:0/1800000510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56191d744060 0x56191d749b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 >> 172.21.15.78:0/1800000510 conn(0x56191d7405f0 msgr2=0x56191d745ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.708+0000 7fb2c5ffb640 0 client.16119 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:34.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.710+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 shutdown_connections 2024-09-16T01:16:34.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.710+0000 7fb2e685b740 1 -- 172.21.15.78:0/1800000510 wait complete. 2024-09-16T01:16:34.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.710+0000 7fb2e685b740 20 client.16119 trim_cache size 0 max 16384 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 24, 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "527487c": { 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 24, 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:34.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 145.3, 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:34.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:34.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:16:34.714 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 msgr2=0x7fb2c80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 0x7fb2c80a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb2d409be50 tx=0x7fb2d40bc040 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 msgr2=0x7fb2dc19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400cad0 tx=0x7fb2d400cf90 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 shutdown_connections 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb2c80a33b0 0x7fb2c80a5870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2dc106a80 0x7fb2dc19ce10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2dc101420 0x7fb2dc198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 --2- 172.21.15.78:0/2154188262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2dc100a80 0x7fb2dc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 >> 172.21.15.78:0/2154188262 conn(0x7fb2dc0fc7a0 msgr2=0x7fb2dc0fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 shutdown_connections 2024-09-16T01:16:34.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:16:34.711+0000 7fb2e4482640 1 -- 172.21.15.78:0/2154188262 wait complete. 2024-09-16T01:16:35.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:35 smithi078 ceph-mon[30937]: pgmap v176: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 629 KiB/s wr, 76 op/s 2024-09-16T01:16:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:35 smithi193 ceph-mon[35701]: pgmap v176: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 629 KiB/s wr, 76 op/s 2024-09-16T01:16:35.551 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '527487c', total seconds elapsed: 28.9 2024-09-16T01:16:35.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:35 smithi148 ceph-mon[35030]: pgmap v176: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 629 KiB/s wr, 76 op/s 2024-09-16T01:16:35.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 320.0 s, 12.2 tps, lat 18353.611 ms stddev 10381.168 2024-09-16T01:16:36.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:36 smithi148 ceph-mon[35030]: pgmap v177: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 766 KiB/s wr, 86 op/s 2024-09-16T01:16:36.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:36 smithi078 ceph-mon[30937]: pgmap v177: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 766 KiB/s wr, 86 op/s 2024-09-16T01:16:36.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:36 smithi193 ceph-mon[35701]: pgmap v177: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 766 KiB/s wr, 86 op/s 2024-09-16T01:16:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:38 smithi078 ceph-mon[30937]: pgmap v178: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 174 KiB/s wr, 20 op/s 2024-09-16T01:16:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:38 smithi193 ceph-mon[35701]: pgmap v178: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 174 KiB/s wr, 20 op/s 2024-09-16T01:16:38.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:38 smithi148 ceph-mon[35030]: pgmap v178: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 174 KiB/s wr, 20 op/s 2024-09-16T01:16:40.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:40 smithi078 ceph-mon[30937]: pgmap v179: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.3 MiB/s wr, 252 op/s 2024-09-16T01:16:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:40 smithi193 ceph-mon[35701]: pgmap v179: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.3 MiB/s wr, 252 op/s 2024-09-16T01:16:40.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:40 smithi148 ceph-mon[35030]: pgmap v179: 129 pgs: 129 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.3 MiB/s wr, 252 op/s 2024-09-16T01:16:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 325.0 s, 1261.5 tps, lat 302.118 ms stddev 2358.965 2024-09-16T01:16:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:42 smithi078 ceph-mon[30937]: pgmap v180: 129 pgs: 129 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.4 MiB/s wr, 319 op/s 2024-09-16T01:16:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:42 smithi193 ceph-mon[35701]: pgmap v180: 129 pgs: 129 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.4 MiB/s wr, 319 op/s 2024-09-16T01:16:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:42 smithi148 ceph-mon[35030]: pgmap v180: 129 pgs: 129 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.4 MiB/s wr, 319 op/s 2024-09-16T01:16:44.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:44 smithi078 ceph-mon[30937]: pgmap v181: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 342 op/s 2024-09-16T01:16:44.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:44 smithi193 ceph-mon[35701]: pgmap v181: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 342 op/s 2024-09-16T01:16:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:44 smithi148 ceph-mon[35030]: pgmap v181: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 342 op/s 2024-09-16T01:16:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 330.0 s, 1383.0 tps, lat 72.703 ms stddev 89.974 2024-09-16T01:16:46.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:46 smithi078 ceph-mon[30937]: pgmap v182: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 544 op/s 2024-09-16T01:16:46.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:46 smithi193 ceph-mon[35701]: pgmap v182: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 544 op/s 2024-09-16T01:16:46.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:46 smithi148 ceph-mon[35030]: pgmap v182: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 17 MiB/s wr, 544 op/s 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 -- 172.21.15.78:0/1425066866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48102480 msgr2=0x7efc4810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 --2- 172.21.15.78:0/1425066866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48102480 0x7efc4810e9b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efc40009ca0 tx=0x7efc4002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 -- 172.21.15.78:0/1425066866 shutdown_connections 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 --2- 172.21.15.78:0/1425066866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48102480 0x7efc4810e9b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 --2- 172.21.15.78:0/1425066866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc48101ae0 0x7efc48101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 --2- 172.21.15.78:0/1425066866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48107b80 0x7efc48107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 -- 172.21.15.78:0/1425066866 >> 172.21.15.78:0/1425066866 conn(0x7efc480fd7e0 msgr2=0x7efc480ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 -- 172.21.15.78:0/1425066866 shutdown_connections 2024-09-16T01:16:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.703+0000 7efc4f578640 1 -- 172.21.15.78:0/1425066866 wait complete. 2024-09-16T01:16:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.704+0000 7efc4f578640 1 Processor -- start 2024-09-16T01:16:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.704+0000 7efc4f578640 1 -- start start 2024-09-16T01:16:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 0x7efc481982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 0x7efc48198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc48107b80 0x7efc4819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4d314640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 0x7efc481982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efc48111f30 con 0x7efc48107b80 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4d314640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 0x7efc481982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47124/0 (socket says 172.21.15.78:47124) 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4d314640 1 -- 172.21.15.78:0/309579080 learned_addr learned my addr 172.21.15.78:0/309579080 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efc48111db0 con 0x7efc48101ae0 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efc481120b0 con 0x7efc48102480 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 0x7efc48198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 -- 172.21.15.78:0/309579080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 msgr2=0x7efc481982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 0x7efc481982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 -- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc48107b80 msgr2=0x7efc4819cb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:16:47.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc48107b80 0x7efc4819cb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.705+0000 7efc4cb13640 1 -- 172.21.15.78:0/309579080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4819d2b0 con 0x7efc48102480 2024-09-16T01:16:47.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.706+0000 7efc4cb13640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 0x7efc48198800 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efc3800e9f0 tx=0x7efc3800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.707+0000 7efc3e7fc640 1 -- 172.21.15.78:0/309579080 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3800ccc0 con 0x7efc48102480 2024-09-16T01:16:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.708+0000 7efc3e7fc640 1 -- 172.21.15.78:0/309579080 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc38010910 con 0x7efc48102480 2024-09-16T01:16:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.708+0000 7efc3e7fc640 1 -- 172.21.15.78:0/309579080 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc380188e0 con 0x7efc48102480 2024-09-16T01:16:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.708+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4819d5a0 con 0x7efc48102480 2024-09-16T01:16:47.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.708+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc48074420 con 0x7efc48102480 2024-09-16T01:16:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.711+0000 7efc3e7fc640 1 -- 172.21.15.78:0/309579080 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7efc38010ac0 con 0x7efc48102480 2024-09-16T01:16:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.711+0000 7efc523f0740 1 Processor -- start 2024-09-16T01:16:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 -- start start 2024-09-16T01:16:47.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5640bb00c2f0 con 0x5640bb0a8410 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5640bafb4270 con 0x5640bb0a7af0 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc523f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640bb0082b0 con 0x5640bb0b2cb0 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc4db15640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc4db15640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47130/0 (socket says 172.21.15.78:47130) 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc4db15640 1 -- 172.21.15.78:0/300044175 learned_addr learned my addr 172.21.15.78:0/300044175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc4d314640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.712+0000 7efc3e7fc640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 0x7efc340a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4cb13640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 -- 172.21.15.78:0/300044175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 msgr2=0x5640bb0b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 -- 172.21.15.78:0/300044175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 msgr2=0x5640bb0b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc3e7fc640 1 -- 172.21.15.78:0/309579080 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efc380d0540 con 0x7efc48102480 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 -- 172.21.15.78:0/300044175 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640bafb3340 con 0x5640bb0a7af0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4d314640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 0x7efc340a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc40005cc0 tx=0x7efc40005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a229c1383e0dda9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.713+0000 7efc4db15640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 0x7efc340a5b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc40041000 tx=0x7efc40043000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc337fe640 1 -- 172.21.15.78:0/300044175 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc400318a0 con 0x5640bb0a7af0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc337fe640 1 -- 172.21.15.78:0/300044175 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc40036040 con 0x5640bb0a7af0 2024-09-16T01:16:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc337fe640 1 -- 172.21.15.78:0/300044175 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40030980 con 0x5640bb0a7af0 2024-09-16T01:16:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc523f0740 1 -- 172.21.15.78:0/300044175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 msgr2=0x5640bb0a7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc523f0740 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc40005cc0 tx=0x7efc40005ef0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc523f0740 1 -- 172.21.15.78:0/300044175 shutdown_connections 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.714+0000 7efc523f0740 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 --2- 172.21.15.78:0/300044175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0a7ed0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 -- 172.21.15.78:0/300044175 >> 172.21.15.78:0/300044175 conn(0x5640bb0a4060 msgr2=0x5640bb0a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 -- 172.21.15.78:0/300044175 shutdown_connections 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 -- 172.21.15.78:0/300044175 wait complete. 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 -- start start 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.715+0000 7efc523f0740 1 Processor -- start 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5640bb0b56f0 con 0x5640bb0a8410 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5640baf95290 con 0x5640bb0a7af0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640bb0b5870 con 0x5640bb0b2cb0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc523f0740 10 client.0 _ll_register_callbacks cb 0x5640bafc6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55756/0 (socket says 172.21.15.78:55756) 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 -- 172.21.15.78:0/42434339 learned_addr learned my addr 172.21.15.78:0/42434339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4d314640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4db15640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 -- 172.21.15.78:0/42434339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 msgr2=0x5640bb0b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 -- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 msgr2=0x5640bb0afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4cb13640 1 -- 172.21.15.78:0/42434339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640bad1f210 con 0x5640bb0a8410 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.716+0000 7efc4d314640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc4cb13640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efc3809c130 tx=0x7efc3809bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7f47492a474fb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc380d0330 con 0x5640bb0a8410 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efc3809b880 con 0x5640bb0a8410 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc40036040 con 0x5640bb0a8410 2024-09-16T01:16:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc3809c430 con 0x5640bb0a8410 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc523f0740 10 client.16128 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.717+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640badeb620 con 0x5640bb0a8410 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc40036040 con 0x5640bb0a8410 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efc38098250 con 0x5640bb0a8410 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 10 client.16128 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 20 client.16128 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb0c2fc0 0x5640bb0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 4 client.16128 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640bb019a20 con 0x5640bb0c2fc0 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.718+0000 7efc4d314640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb0c2fc0 0x5640bb0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.719+0000 7efc30ff9640 10 client.16128 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.720+0000 7efc4d314640 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb0c2fc0 0x5640bb0c53f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad4e76d13fb0d0d server_cookie=de63780daf3352a7 in_seq=0 out_seq=0 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc30ff9640 1 -- 172.21.15.78:0/42434339 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640bb019a20 con 0x5640bb0c2fc0 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc30ff9640 10 client.16128 handle_command_reply: tid=0 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc523f0740 1 client.16128 shutdown 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 msgr2=0x5640bb0b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efc3809c130 tx=0x7efc3809bf70 comp rx=0 tx=0).stop 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 shutdown_connections 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.721+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb0c2fc0 0x5640bb0c53f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b0170 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 --2- 172.21.15.78:0/42434339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 >> 172.21.15.78:0/42434339 conn(0x5640bb0a4060 msgr2=0x5640bb0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc30ff9640 0 client.16128 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 shutdown_connections 2024-09-16T01:16:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 1 -- 172.21.15.78:0/42434339 wait complete. 2024-09-16T01:16:47.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.722+0000 7efc523f0740 20 client.16128 trim_cache size 0 max 16384 2024-09-16T01:16:47.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.743+0000 7efc523f0740 1 Processor -- start 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.743+0000 7efc523f0740 1 -- start start 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efc481b3a30 con 0x5640bb0a8410 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efc481b1300 con 0x5640bb0a7af0 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc523f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efc481af150 con 0x5640bb0b2cb0 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4d314640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54340/0 (socket says 172.21.15.78:54340) 2024-09-16T01:16:47.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 -- 172.21.15.78:0/1795672243 learned_addr learned my addr 172.21.15.78:0/1795672243 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4cb13640 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 -- 172.21.15.78:0/1795672243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 msgr2=0x5640bb0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 -- 172.21.15.78:0/1795672243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 msgr2=0x5640bb0b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 -- 172.21.15.78:0/1795672243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640bb09f2d0 con 0x5640bb0b2cb0 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.744+0000 7efc4db15640 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc40004960 tx=0x7efc40004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef62601b89a1a222 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.745+0000 7efc32ffd640 1 -- 172.21.15.78:0/1795672243 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40058020 con 0x5640bb0b2cb0 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.745+0000 7efc32ffd640 1 -- 172.21.15.78:0/1795672243 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc40036070 con 0x5640bb0b2cb0 2024-09-16T01:16:47.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.745+0000 7efc32ffd640 1 -- 172.21.15.78:0/1795672243 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4002fb50 con 0x5640bb0b2cb0 2024-09-16T01:16:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.745+0000 7efc523f0740 1 -- 172.21.15.78:0/1795672243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 msgr2=0x5640bb0cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.745+0000 7efc523f0740 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc40004960 tx=0x7efc40004990 comp rx=0 tx=0).stop 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 -- 172.21.15.78:0/1795672243 shutdown_connections 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0b2cb0 0x5640bb0cad00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 --2- 172.21.15.78:0/1795672243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0a7af0 0x5640bb0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 -- 172.21.15.78:0/1795672243 >> 172.21.15.78:0/1795672243 conn(0x5640bb0a4060 msgr2=0x5640bb0b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.746+0000 7efc523f0740 1 -- 172.21.15.78:0/1795672243 shutdown_connections 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.747+0000 7efc523f0740 1 -- 172.21.15.78:0/1795672243 wait complete. 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.747+0000 7efc523f0740 1 -- start start 2024-09-16T01:16:47.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.747+0000 7efc523f0740 1 Processor -- start 2024-09-16T01:16:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efc481af7c0 con 0x5640bb0a8410 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efc481b0c30 con 0x5640bb0b2cb0 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efc481afe90 con 0x5640bb0a7af0 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc523f0740 10 client.0 _ll_register_callbacks cb 0x7efc3800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc4d314640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc4d314640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:54346/0 (socket says 172.21.15.78:54346) 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc4d314640 1 -- 172.21.15.78:0/2470075408 learned_addr learned my addr 172.21.15.78:0/2470075408 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.748+0000 7efc4cb13640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 -- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0b2cb0 msgr2=0x5640bb0b6540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b6540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 -- 172.21.15.78:0/2470075408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 msgr2=0x5640bb0b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:16:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 -- 172.21.15.78:0/2470075408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640bb12bf10 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc4d314640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5640bb0c5c70 tx=0x7efc44012b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b334e765dfb6da7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4402a020 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7efc44004250 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc40036070 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.749+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc44032050 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 10 client.25585 fetch_fsmap learned FSMap version 31 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640bb12d260 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc40036070 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efc44025070 con 0x5640bb0a7af0 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 10 client.25585 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:16:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 10 client.25585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:16:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 20 client.25585 populate_metadata read hostname 'smithi078' 2024-09-16T01:16:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb12da00 0x5640bb12fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:16:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 4 client.25585 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:16:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640bb013d70 con 0x5640bb12da00 2024-09-16T01:16:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.750+0000 7efc4cb13640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb12da00 0x5640bb12fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:16:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.751+0000 7efc30ff9640 10 client.25585 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.751+0000 7efc4cb13640 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb12da00 0x5640bb12fe60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e451b09dcd1b8a8 server_cookie=749c44abac515bee in_seq=0 out_seq=0 2024-09-16T01:16:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.758+0000 7efc30ff9640 1 -- 172.21.15.78:0/2470075408 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640bb013d70 con 0x5640bb12da00 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.758+0000 7efc30ff9640 10 client.25585 handle_command_reply: tid=0 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.758+0000 7efc523f0740 1 client.25585 shutdown 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 msgr2=0x5640bb0b5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x5640bb0c5c70 tx=0x7efc44012b40 comp rx=0 tx=0).stop 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 shutdown_connections 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640bb12da00 0x5640bb12fe60 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640bb0b2cb0 0x5640bb0b6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640bb0a8410 0x5640bb0b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 --2- 172.21.15.78:0/2470075408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640bb0a7af0 0x5640bb0b5ac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 >> 172.21.15.78:0/2470075408 conn(0x5640bb0a4060 msgr2=0x5640bb0ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc30ff9640 0 client.25585 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 shutdown_connections 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 1 -- 172.21.15.78:0/2470075408 wait complete. 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc523f0740 20 client.25585 trim_cache size 0 max 16384 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:16:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:16:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 msgr2=0x7efc340a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 0x7efc340a5b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc40041000 tx=0x7efc40043000 comp rx=0 tx=0).stop 2024-09-16T01:16:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 msgr2=0x7efc48198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.759+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 0x7efc48198800 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efc3800e9f0 tx=0x7efc3800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 shutdown_connections 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efc340a3640 0x7efc340a5b00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc48107b80 0x7efc4819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efc48102480 0x7efc48198800 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 --2- 172.21.15.78:0/309579080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc48101ae0 0x7efc481982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 >> 172.21.15.78:0/309579080 conn(0x7efc480fd7e0 msgr2=0x7efc480fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:16:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 shutdown_connections 2024-09-16T01:16:47.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:16:47.760+0000 7efc4f578640 1 -- 172.21.15.78:0/309579080 wait complete. 2024-09-16T01:16:48.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:48 smithi078 ceph-mon[30937]: pgmap v183: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 17 MiB/s wr, 556 op/s 2024-09-16T01:16:48.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:48 smithi193 ceph-mon[35701]: pgmap v183: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 17 MiB/s wr, 556 op/s 2024-09-16T01:16:48.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:48 smithi148 ceph-mon[35030]: pgmap v183: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 17 MiB/s wr, 556 op/s 2024-09-16T01:16:50.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:50 smithi078 ceph-mon[30937]: pgmap v184: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 MiB/s wr, 698 op/s 2024-09-16T01:16:50.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:50 smithi193 ceph-mon[35701]: pgmap v184: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 MiB/s wr, 698 op/s 2024-09-16T01:16:50.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:50 smithi148 ceph-mon[35030]: pgmap v184: 129 pgs: 129 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 MiB/s wr, 698 op/s 2024-09-16T01:16:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 335.0 s, 1396.6 tps, lat 71.520 ms stddev 86.022 2024-09-16T01:16:52.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:52 smithi078 ceph-mon[30937]: pgmap v185: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 587 op/s 2024-09-16T01:16:52.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:52 smithi193 ceph-mon[35701]: pgmap v185: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 587 op/s 2024-09-16T01:16:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:52 smithi148 ceph-mon[35030]: pgmap v185: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 21 MiB/s wr, 587 op/s 2024-09-16T01:16:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:54 smithi078 ceph-mon[30937]: pgmap v186: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 19 MiB/s wr, 527 op/s 2024-09-16T01:16:54.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:54 smithi193 ceph-mon[35701]: pgmap v186: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 19 MiB/s wr, 527 op/s 2024-09-16T01:16:54.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:54 smithi148 ceph-mon[35030]: pgmap v186: 129 pgs: 129 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 19 MiB/s wr, 527 op/s 2024-09-16T01:16:55.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:55 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:16:55.786 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:55 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:16:55.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:55 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:16:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 340.0 s, 1501.8 tps, lat 66.867 ms stddev 84.461 2024-09-16T01:16:56.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:56 smithi078 ceph-mon[30937]: pgmap v187: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 709 op/s 2024-09-16T01:16:56.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:56 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:16:56.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:56 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:16:56.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:56 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:16:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:56 smithi193 ceph-mon[35701]: pgmap v187: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 709 op/s 2024-09-16T01:16:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:56 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:16:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:56 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:16:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:56 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:16:56.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:56 smithi148 ceph-mon[35030]: pgmap v187: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 24 MiB/s wr, 709 op/s 2024-09-16T01:16:56.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:56 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:16:56.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:56 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:16:56.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:56 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:16:58.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:16:58 smithi193 ceph-mon[35701]: pgmap v188: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 537 op/s 2024-09-16T01:16:58.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:16:58 smithi148 ceph-mon[35030]: pgmap v188: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 537 op/s 2024-09-16T01:16:59.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:16:58 smithi078 ceph-mon[30937]: pgmap v188: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 19 MiB/s wr, 537 op/s 2024-09-16T01:17:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:00 smithi193 ceph-mon[35701]: pgmap v189: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 23 MiB/s wr, 627 op/s 2024-09-16T01:17:00.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:00 smithi148 ceph-mon[35030]: pgmap v189: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 23 MiB/s wr, 627 op/s 2024-09-16T01:17:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.915+0000 7f53a2f0c640 1 -- 172.21.15.78:0/4293911831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c100780 msgr2=0x7f539c109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.915+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/4293911831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c100780 0x7f539c109b40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f538c009860 tx=0x7f538c02a380 comp rx=0 tx=0).stop 2024-09-16T01:17:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.920+0000 7f53a2f0c640 1 -- 172.21.15.78:0/4293911831 shutdown_connections 2024-09-16T01:17:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.920+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/4293911831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c10a2a0 0x7f539c110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.920+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/4293911831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c100780 0x7f539c109b40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.920+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/4293911831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f539c0ffdd0 0x7f539c1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.920+0000 7f53a2f0c640 1 -- 172.21.15.78:0/4293911831 >> 172.21.15.78:0/4293911831 conn(0x7f539c0fba30 msgr2=0x7f539c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.926+0000 7f53a2f0c640 1 -- 172.21.15.78:0/4293911831 shutdown_connections 2024-09-16T01:17:00.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.929+0000 7f53a2f0c640 1 -- 172.21.15.78:0/4293911831 wait complete. 2024-09-16T01:17:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 Processor -- start 2024-09-16T01:17:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 -- start start 2024-09-16T01:17:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f539c0ffdd0 0x7f539c11a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c1105b0 0x7f539c110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f539c105470 con 0x7f539c10a2a0 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f539c1052f0 con 0x7f539c1105b0 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.930+0000 7f53a2f0c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f539c1055f0 con 0x7f539c0ffdd0 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46362/0 (socket says 172.21.15.78:46362) 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 -- 172.21.15.78:0/1073940233 learned_addr learned my addr 172.21.15.78:0/1073940233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f53a14a9640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c1105b0 0x7f539c110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 -- 172.21.15.78:0/1073940233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f539c0ffdd0 msgr2=0x7f539c11a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f539c0ffdd0 0x7f539c11a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 -- 172.21.15.78:0/1073940233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c1105b0 msgr2=0x7f539c110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c1105b0 0x7f539c110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.931+0000 7f539bfff640 1 -- 172.21.15.78:0/1073940233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c115d60 con 0x7f539c10a2a0 2024-09-16T01:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.932+0000 7f539bfff640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f538c035040 tx=0x7f538c030f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.932+0000 7f5399ffb640 1 -- 172.21.15.78:0/1073940233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c037040 con 0x7f539c10a2a0 2024-09-16T01:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.932+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c115ff0 con 0x7f539c10a2a0 2024-09-16T01:17:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.933+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c1164e0 con 0x7f539c10a2a0 2024-09-16T01:17:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.933+0000 7f5399ffb640 1 -- 172.21.15.78:0/1073940233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f538c062070 con 0x7f539c10a2a0 2024-09-16T01:17:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.936+0000 7f5399ffb640 1 -- 172.21.15.78:0/1073940233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c0046c0 con 0x7f539c10a2a0 2024-09-16T01:17:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.936+0000 7f5399ffb640 1 -- 172.21.15.78:0/1073940233 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f538c004920 con 0x7f539c10a2a0 2024-09-16T01:17:00.939 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 345.0 s, 1225.6 tps, lat 79.011 ms stddev 91.499 2024-09-16T01:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.940+0000 7f5399ffb640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 0x7f53840a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.940+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 0x7f53840a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.941+0000 7f5399ffb640 1 -- 172.21.15.78:0/1073940233 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f538c112a30 con 0x7f539c10a2a0 2024-09-16T01:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 0x7f53840a5b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f5390009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 Processor -- start 2024-09-16T01:17:00.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 -- start start 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f17ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f22770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f250a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562492e7c2f0 con 0x562492f18410 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562492e24270 con 0x562492f22cb0 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.944+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562492e782b0 con 0x562492f17af0 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a14a9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f250a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f17ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a14a9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f250a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43138/0 (socket says 172.21.15.78:43138) 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a14a9640 1 -- 172.21.15.78:0/3696274863 learned_addr learned my addr 172.21.15.78:0/3696274863 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f539bfff640 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f22770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 -- 172.21.15.78:0/3696274863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 msgr2=0x562492f250a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f250a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 -- 172.21.15.78:0/3696274863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 msgr2=0x562492f22770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f22770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.945+0000 7f53a0ca8640 1 -- 172.21.15.78:0/3696274863 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562492e23340 con 0x562492f17af0 2024-09-16T01:17:00.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.946+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f17ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f539000ed30 tx=0x7f5390011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b57f84f4b9a74d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.950+0000 7f5382ffd640 1 -- 172.21.15.78:0/3696274863 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390020070 con 0x562492f17af0 2024-09-16T01:17:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.950+0000 7f5382ffd640 1 -- 172.21.15.78:0/3696274863 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5390010e40 con 0x562492f17af0 2024-09-16T01:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.951+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3696274863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 msgr2=0x562492f17ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.951+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f17ed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f539000ed30 tx=0x7f5390011000 comp rx=0 tx=0).stop 2024-09-16T01:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.952+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3696274863 shutdown_connections 2024-09-16T01:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.952+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f250a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.952+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f22770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.952+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3696274863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f17ed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.952+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3696274863 >> 172.21.15.78:0/3696274863 conn(0x562492f14060 msgr2=0x562492f14440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.955+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3696274863 shutdown_connections 2024-09-16T01:17:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.956+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3696274863 wait complete. 2024-09-16T01:17:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.956+0000 7f53a5c9d740 1 -- start start 2024-09-16T01:17:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.956+0000 7f53a5c9d740 1 Processor -- start 2024-09-16T01:17:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.957+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f1fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.957+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f18410 0x562492f1fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.957+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562492f258d0 con 0x562492f17af0 2024-09-16T01:17:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562492e05290 con 0x562492f18410 2024-09-16T01:17:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a14a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a14a9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:39712/0 (socket says 172.21.15.78:39712) 2024-09-16T01:17:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a14a9640 1 -- 172.21.15.78:0/3626739423 learned_addr learned my addr 172.21.15.78:0/3626739423 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562492f25a50 con 0x562492f22cb0 2024-09-16T01:17:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.958+0000 7f53a5c9d740 10 client.0 _ll_register_callbacks cb 0x562492e36140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 -- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f18410 msgr2=0x562492f1fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f18410 0x562492f1fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 -- 172.21.15.78:0/3626739423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 msgr2=0x562492f1fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f1fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 -- 172.21.15.78:0/3626739423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562492b8f210 con 0x562492f22cb0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.960+0000 7f53a14a9640 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5394003f00 tx=0x7f5394004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c07a539cbc2896a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f537bfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394012a60 con 0x562492f22cb0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562492e89a20 con 0x562492f22cb0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f5394012c00 con 0x562492f22cb0 2024-09-16T01:17:00.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5394012e00 con 0x562492f22cb0 2024-09-16T01:17:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f53a5c9d740 10 client.25594 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.961+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562492c5b620 con 0x562492f22cb0 2024-09-16T01:17:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.963+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562492e89a20 con 0x562492f22cb0 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.963+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5394019070 con 0x562492f22cb0 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 10 client.25594 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 10 client.25594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 20 client.25594 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f33120 0x562492f355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 4 client.25594 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:17:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562492e83d70 con 0x562492f33120 2024-09-16T01:17:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.964+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f33120 0x562492f355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.966+0000 7f537bfff640 10 client.25594 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.966+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f33120 0x562492f355e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8e7f2c7846e208 server_cookie=719a7e2ae53b7e8 in_seq=0 out_seq=0 2024-09-16T01:17:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.971+0000 7f537bfff640 1 -- 172.21.15.78:0/3626739423 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562492e83d70 con 0x562492f33120 2024-09-16T01:17:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.971+0000 7f537bfff640 10 client.25594 handle_command_reply: tid=0 2024-09-16T01:17:00.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.971+0000 7f53a5c9d740 1 client.25594 shutdown 2024-09-16T01:17:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.972+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 msgr2=0x562492f20530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.972+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5394003f00 tx=0x7f5394004410 comp rx=0 tx=0).stop 2024-09-16T01:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 shutdown_connections 2024-09-16T01:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f33120 0x562492f355e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f22cb0 0x562492f20530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f18410 0x562492f1fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3626739423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f1fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.974+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 >> 172.21.15.78:0/3626739423 conn(0x562492f14060 msgr2=0x562492f19dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.976+0000 7f537bfff640 0 client.25594 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.980+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 shutdown_connections 2024-09-16T01:17:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.984+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3626739423 wait complete. 2024-09-16T01:17:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:00.984+0000 7f53a5c9d740 20 client.25594 trim_cache size 0 max 16384 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 Processor -- start 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 -- start start 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f20360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f3b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5388012910 con 0x562492f18410 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f53880101e0 con 0x562492f22cb0 2024-09-16T01:17:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.017+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f538800e030 con 0x562492f17af0 2024-09-16T01:17:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.018+0000 7f539bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.018+0000 7f539bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46388/0 (socket says 172.21.15.78:46388) 2024-09-16T01:17:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.018+0000 7f539bfff640 1 -- 172.21.15.78:0/2027453587 learned_addr learned my addr 172.21.15.78:0/2027453587 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.018+0000 7f53a14a9640 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f3b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.020+0000 7f539bfff640 1 -- 172.21.15.78:0/2027453587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 msgr2=0x562492f20360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:17:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.020+0000 7f539bfff640 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f20360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.020+0000 7f539bfff640 1 -- 172.21.15.78:0/2027453587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 msgr2=0x562492f3b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.020+0000 7f539bfff640 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f3b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.020+0000 7f539bfff640 1 -- 172.21.15.78:0/2027453587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562492f13e90 con 0x562492f18410 2024-09-16T01:17:01.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.024+0000 7f539bfff640 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f538c0dafa0 tx=0x7f538c0de790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f8fe346610bafa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:01.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.027+0000 7f53827fc640 1 -- 172.21.15.78:0/2027453587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c0df240 con 0x562492f18410 2024-09-16T01:17:01.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.027+0000 7f53827fc640 1 -- 172.21.15.78:0/2027453587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f538c0deb40 con 0x562492f18410 2024-09-16T01:17:01.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.027+0000 7f53827fc640 1 -- 172.21.15.78:0/2027453587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c004240 con 0x562492f18410 2024-09-16T01:17:01.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:00 smithi078 ceph-mon[30937]: pgmap v189: 129 pgs: 129 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 23 MiB/s wr, 627 op/s 2024-09-16T01:17:01.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.029+0000 7f53a5c9d740 1 -- 172.21.15.78:0/2027453587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 msgr2=0x562492f3aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:01.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.030+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f538c0dafa0 tx=0x7f538c0de790 comp rx=0 tx=0).stop 2024-09-16T01:17:01.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.034+0000 7f53a5c9d740 1 -- 172.21.15.78:0/2027453587 shutdown_connections 2024-09-16T01:17:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.034+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f3b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.034+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f18410 0x562492f3aef0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.034+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/2027453587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f17af0 0x562492f20360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.034+0000 7f53a5c9d740 1 -- 172.21.15.78:0/2027453587 >> 172.21.15.78:0/2027453587 conn(0x562492f14060 msgr2=0x562492f1eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.036+0000 7f53a5c9d740 1 -- 172.21.15.78:0/2027453587 shutdown_connections 2024-09-16T01:17:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.036+0000 7f53a5c9d740 1 -- 172.21.15.78:0/2027453587 wait complete. 2024-09-16T01:17:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.037+0000 7f53a5c9d740 1 -- start start 2024-09-16T01:17:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.037+0000 7f53a5c9d740 1 Processor -- start 2024-09-16T01:17:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.037+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f26500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f95e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f538800e6a0 con 0x562492f17af0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f538800fb10 con 0x562492f22cb0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f538800ed70 con 0x562492f18410 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a5c9d740 10 client.0 _ll_register_callbacks cb 0x7f5394009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f539bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f539bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:39728/0 (socket says 172.21.15.78:39728) 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f539bfff640 1 -- 172.21.15.78:0/3331549385 learned_addr learned my addr 172.21.15.78:0/3331549385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.038+0000 7f53a14a9640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f95e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.039+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f26500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.039+0000 7f539bfff640 1 -- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 msgr2=0x562492f95e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.039+0000 7f539bfff640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f95e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.039+0000 7f539bfff640 1 -- 172.21.15.78:0/3331549385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 msgr2=0x562492f26500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.040+0000 7f539bfff640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f26500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.040+0000 7f539bfff640 1 -- 172.21.15.78:0/3331549385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562492f96470 con 0x562492f18410 2024-09-16T01:17:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.044+0000 7f539bfff640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f538c0de840 tx=0x7f538c110240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efda7b472240bfb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.044+0000 7f537bfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:17:01.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.048+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c0df240 con 0x562492f18410 2024-09-16T01:17:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.048+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f538c0deb40 con 0x562492f18410 2024-09-16T01:17:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.048+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9437+0+0 (secure 0 0 0) 0x7f538c004240 con 0x562492f18410 2024-09-16T01:17:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.055+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f538c004440 con 0x562492f18410 2024-09-16T01:17:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.055+0000 7f53a5c9d740 10 client.25597 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.056+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562492f9d7e0 con 0x562492f18410 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f538c0deb40 con 0x562492f18410 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f538c0ef030 con 0x562492f18410 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 10 client.25597 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 20 client.25597 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f9df40 0x562492fa0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 4 client.25597 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:17:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562492e00900 con 0x562492f9df40 2024-09-16T01:17:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.058+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f9df40 0x562492fa0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:01.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.064+0000 7f53a0ca8640 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f9df40 0x562492fa0360 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843fc1e4daa84cbf server_cookie=2a2e41985cf0ae26 in_seq=0 out_seq=0 2024-09-16T01:17:01.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:01.064+0000 7f537bfff640 10 client.25597 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f537bfff640 1 -- 172.21.15.78:0/3331549385 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562492e00900 con 0x562492f9df40 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f537bfff640 10 client.25597 handle_command_reply: tid=0 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 client.25597 shutdown 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 msgr2=0x562492f26a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f538c0de840 tx=0x7f538c110240 comp rx=0 tx=0).stop 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 shutdown_connections 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562492f9df40 0x562492fa0360 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562492f22cb0 0x562492f95e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562492f18410 0x562492f26a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f537bfff640 0 client.25597 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 --2- 172.21.15.78:0/3331549385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562492f17af0 0x562492f26500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.788+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 >> 172.21.15.78:0/3331549385 conn(0x562492f14060 msgr2=0x562492f20370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.789+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 shutdown_connections 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.789+0000 7f53a5c9d740 1 -- 172.21.15.78:0/3331549385 wait complete. 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.789+0000 7f53a5c9d740 20 client.25597 trim_cache size 0 max 16384 2024-09-16T01:17:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:17:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:17:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:17:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 msgr2=0x7f53840a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 0x7f53840a5b50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5390005fd0 tx=0x7f5390009260 comp rx=0 tx=0).stop 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 msgr2=0x7f539c110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f538c035040 tx=0x7f538c030f70 comp rx=0 tx=0).stop 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 shutdown_connections 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f53840a3690 0x7f53840a5b50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f539c1105b0 0x7f539c110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f539c10a2a0 0x7f539c110070 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.790+0000 7f53a2f0c640 1 --2- 172.21.15.78:0/1073940233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f539c0ffdd0 0x7f539c11a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:02.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.791+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 >> 172.21.15.78:0/1073940233 conn(0x7f539c0fba30 msgr2=0x7f539c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:02.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.791+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 shutdown_connections 2024-09-16T01:17:02.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:02.791+0000 7f53a2f0c640 1 -- 172.21.15.78:0/1073940233 wait complete. 2024-09-16T01:17:02.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:02 smithi193 ceph-mon[35701]: pgmap v190: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 MiB/s wr, 639 op/s 2024-09-16T01:17:02.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:02 smithi148 ceph-mon[35030]: pgmap v190: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 MiB/s wr, 639 op/s 2024-09-16T01:17:03.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:02 smithi078 ceph-mon[30937]: pgmap v190: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 MiB/s wr, 639 op/s 2024-09-16T01:17:04.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:04 smithi148 ceph-mon[35030]: pgmap v191: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 570 op/s 2024-09-16T01:17:04.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:04 smithi148 ceph-mon[35030]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T01:17:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:04 smithi078 ceph-mon[30937]: pgmap v191: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 570 op/s 2024-09-16T01:17:05.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:04 smithi078 ceph-mon[30937]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T01:17:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:04 smithi193 ceph-mon[35701]: pgmap v191: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 17 MiB/s wr, 570 op/s 2024-09-16T01:17:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:04 smithi193 ceph-mon[35701]: osdmap e95: 12 total, 12 up, 12 in 2024-09-16T01:17:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 350.0 s, 1023.3 tps, lat 100.816 ms stddev 212.583 2024-09-16T01:17:06.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:06 smithi148 ceph-mon[35030]: pgmap v193: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.44k op/s 2024-09-16T01:17:07.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:06 smithi078 ceph-mon[30937]: pgmap v193: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.44k op/s 2024-09-16T01:17:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:06 smithi193 ceph-mon[35701]: pgmap v193: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.44k op/s 2024-09-16T01:17:08.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:08 smithi078 ceph-mon[30937]: pgmap v194: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.56k op/s 2024-09-16T01:17:08.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:08 smithi148 ceph-mon[35030]: pgmap v194: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.56k op/s 2024-09-16T01:17:09.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:08 smithi193 ceph-mon[35701]: pgmap v194: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.56k op/s 2024-09-16T01:17:10.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:10 smithi148 ceph-mon[35030]: pgmap v195: 129 pgs: 129 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-09-16T01:17:10.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:10 smithi148 ceph-mon[35030]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T01:17:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 355.0 s, 1692.6 tps, lat 59.215 ms stddev 73.713 2024-09-16T01:17:11.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:10 smithi078 ceph-mon[30937]: pgmap v195: 129 pgs: 129 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-09-16T01:17:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:10 smithi078 ceph-mon[30937]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T01:17:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:10 smithi193 ceph-mon[35701]: pgmap v195: 129 pgs: 129 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-09-16T01:17:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:10 smithi193 ceph-mon[35701]: osdmap e96: 12 total, 12 up, 12 in 2024-09-16T01:17:12.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:12 smithi148 ceph-mon[35030]: pgmap v197: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 32 MiB/s wr, 2.03k op/s 2024-09-16T01:17:13.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:12 smithi078 ceph-mon[30937]: pgmap v197: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 32 MiB/s wr, 2.03k op/s 2024-09-16T01:17:13.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:12 smithi193 ceph-mon[35701]: pgmap v197: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 32 MiB/s wr, 2.03k op/s 2024-09-16T01:17:14.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:14 smithi148 ceph-mon[35030]: pgmap v198: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.47k op/s 2024-09-16T01:17:15.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:14 smithi078 ceph-mon[30937]: pgmap v198: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.47k op/s 2024-09-16T01:17:15.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:14 smithi193 ceph-mon[35701]: pgmap v198: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 26 MiB/s wr, 1.47k op/s 2024-09-16T01:17:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.931+0000 7fe2bc95f640 1 -- 172.21.15.78:0/2089203372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b8101370 msgr2=0x7fe2b810d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.931+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/2089203372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b8101370 0x7fe2b810d8a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009cd0 tx=0x7fe2b002a910 comp rx=0 tx=0).stop 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.932+0000 7fe2bc95f640 1 -- 172.21.15.78:0/2089203372 shutdown_connections 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.932+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/2089203372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b8101370 0x7fe2b810d8a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.932+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/2089203372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2b81009d0 0x7fe2b8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.932+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/2089203372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81069e0 0x7fe2b8106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.932+0000 7fe2bc95f640 1 -- 172.21.15.78:0/2089203372 >> 172.21.15.78:0/2089203372 conn(0x7fe2b80fc6f0 msgr2=0x7fe2b80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:15.938 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 360.0 s, 1539.5 tps, lat 64.017 ms stddev 72.742 2024-09-16T01:17:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.937+0000 7fe2bc95f640 1 -- 172.21.15.78:0/2089203372 shutdown_connections 2024-09-16T01:17:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.938+0000 7fe2bc95f640 1 -- 172.21.15.78:0/2089203372 wait complete. 2024-09-16T01:17:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.939+0000 7fe2bc95f640 1 Processor -- start 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 -- start start 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81009d0 0x7fe2b819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2b8101370 0x7fe2b819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8111f30 con 0x7fe2b81009d0 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8111db0 con 0x7fe2b8101370 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.941+0000 7fe2bc95f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81120b0 con 0x7fe2b81069e0 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:57466/0 (socket says 172.21.15.78:57466) 2024-09-16T01:17:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3360717678 learned_addr learned my addr 172.21.15.78:0/3360717678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b659c640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81009d0 0x7fe2b819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3360717678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2b8101370 msgr2=0x7fe2b819f520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:17:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.942+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2b8101370 0x7fe2b819f520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.943+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81009d0 msgr2=0x7fe2b819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.943+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81009d0 0x7fe2b819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.943+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3360717678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b810b0f0 con 0x7fe2b81069e0 2024-09-16T01:17:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.943+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0005e30 tx=0x7fe2b0030d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.946+0000 7fe2a77fe640 1 -- 172.21.15.78:0/3360717678 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0003970 con 0x7fe2b81069e0 2024-09-16T01:17:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.946+0000 7fe2a77fe640 1 -- 172.21.15.78:0/3360717678 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b0031460 con 0x7fe2b81069e0 2024-09-16T01:17:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.946+0000 7fe2a77fe640 1 -- 172.21.15.78:0/3360717678 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0003970 con 0x7fe2b81069e0 2024-09-16T01:17:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.947+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b810b380 con 0x7fe2b81069e0 2024-09-16T01:17:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.947+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b819ac00 con 0x7fe2b81069e0 2024-09-16T01:17:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.951+0000 7fe2bf7d7740 1 Processor -- start 2024-09-16T01:17:15.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.953+0000 7fe2a77fe640 1 -- 172.21.15.78:0/3360717678 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fe2b0003b10 con 0x7fe2b81069e0 2024-09-16T01:17:15.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.954+0000 7fe2a77fe640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 0x7fe2a00a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.954+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 0x7fe2a00a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.954+0000 7fe2a77fe640 1 -- 172.21.15.78:0/3360717678 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fe2b0030e70 con 0x7fe2b81069e0 2024-09-16T01:17:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.956+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 0x7fe2a00a5d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8197750 tx=0x7fe2a8006bd0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 -- start start 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 0x55ac96a7eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a89770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ac969e32f0 con 0x55ac96a89cb0 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ac9698b270 con 0x55ac96a7f410 2024-09-16T01:17:15.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.958+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac969df2b0 con 0x55ac96a7eaf0 2024-09-16T01:17:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.960+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 0x55ac96a7eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.960+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 0x55ac96a7eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:57476/0 (socket says 172.21.15.78:57476) 2024-09-16T01:17:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.960+0000 7fe2b659c640 1 -- 172.21.15.78:0/3735892457 learned_addr learned my addr 172.21.15.78:0/3735892457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.960+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.963 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3735892457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 msgr2=0x55ac96a7eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 0x55ac96a7eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3735892457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 msgr2=0x55ac96a89770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:17:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a89770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/3735892457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac9698a340 con 0x55ac96a89cb0 2024-09-16T01:17:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.961+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8c0a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ee450 tx=0x7fe2b00c1580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2848ba245c46e706 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.962+0000 7fe2a4ff9640 1 -- 172.21.15.78:0/3735892457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00b9a10 con 0x55ac96a89cb0 2024-09-16T01:17:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.962+0000 7fe2a4ff9640 1 -- 172.21.15.78:0/3735892457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b00329e0 con 0x55ac96a89cb0 2024-09-16T01:17:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.962+0000 7fe2a4ff9640 1 -- 172.21.15.78:0/3735892457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00b9bb0 con 0x55ac96a89cb0 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3735892457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 msgr2=0x55ac96a8c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8c0a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ee450 tx=0x7fe2b00c1580 comp rx=0 tx=0).stop 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3735892457 shutdown_connections 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8c0a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a89770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3735892457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7eaf0 0x55ac96a7eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.963+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3735892457 >> 172.21.15.78:0/3735892457 conn(0x55ac96a7b060 msgr2=0x55ac96a7b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.967+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3735892457 shutdown_connections 2024-09-16T01:17:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.967+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3735892457 wait complete. 2024-09-16T01:17:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.967+0000 7fe2bf7d7740 1 -- start start 2024-09-16T01:17:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.967+0000 7fe2bf7d7740 1 Processor -- start 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a86cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a87200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a87740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ac96a8c640 con 0x55ac96a89cb0 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ac9696c290 con 0x55ac96a7eaf0 2024-09-16T01:17:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac96a8c7c0 con 0x55ac96a7f410 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2bf7d7740 10 client.0 _ll_register_callbacks cb 0x55ac9699d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a86cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a87200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a86cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44762/0 (socket says 172.21.15.78:44762) 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b659c640 1 -- 172.21.15.78:0/2841978760 learned_addr learned my addr 172.21.15.78:0/2841978760 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a87740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 msgr2=0x55ac96a86cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a86cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/2841978760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 msgr2=0x55ac96a87740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a87740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.968+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/2841978760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac966f6210 con 0x55ac96a7f410 2024-09-16T01:17:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a87200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe2b81977e0 tx=0x7fe2a8004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1727a03b78731d1a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe29dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a801d5f0 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2b00329e0 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fe2a801d790 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2a802b050 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe2bf7d7740 10 client.25609 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ac967c2620 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.969+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b00329e0 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe2a8012070 con 0x55ac96a7f410 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 10 client.25609 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 10 client.25609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 20 client.25609 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96a99ae0 0x55ac96a9bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 4 client.25609 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:17:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac969f0a20 con 0x55ac96a99ae0 2024-09-16T01:17:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.970+0000 7fe2b659c640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96a99ae0 0x55ac96a9bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.971+0000 7fe2b659c640 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96a99ae0 0x55ac96a9bed0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0e92e0c15fac9d6 server_cookie=a9e27562fd8a5311 in_seq=0 out_seq=0 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.971+0000 7fe29dffb640 10 client.25609 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe29dffb640 1 -- 172.21.15.78:0/2841978760 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac969f0a20 con 0x55ac96a99ae0 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe29dffb640 10 client.25609 handle_command_reply: tid=0 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 client.25609 shutdown 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 msgr2=0x55ac96a87200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a87200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe2b81977e0 tx=0x7fe2a8004700 comp rx=0 tx=0).stop 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 shutdown_connections 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96a99ae0 0x55ac96a9bed0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a87740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a87200 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2841978760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 >> 172.21.15.78:0/2841978760 conn(0x55ac96a7b060 msgr2=0x55ac96a80dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe29dffb640 0 client.25609 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.972+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 shutdown_connections 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.973+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2841978760 wait complete. 2024-09-16T01:17:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.973+0000 7fe2bf7d7740 20 client.25609 trim_cache size 0 max 16384 2024-09-16T01:17:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.992+0000 7fe2bf7d7740 1 Processor -- start 2024-09-16T01:17:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.992+0000 7fe2bf7d7740 1 -- start start 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a7eaf0 0x55ac96a88ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a87570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81c8f50 con 0x55ac96a7eaf0 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81c6820 con 0x55ac96a7f410 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2b5d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a87570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:57498/0 (socket says 172.21.15.78:57498) 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/2812306934 learned_addr learned my addr 172.21.15.78:0/2812306934 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.993+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8072e50 con 0x55ac96a89cb0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/2812306934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 msgr2=0x55ac96a87570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a87570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/2812306934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a7eaf0 msgr2=0x55ac96a88ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a7eaf0 0x55ac96a88ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 -- 172.21.15.78:0/2812306934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac96a98610 con 0x55ac96a89cb0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe2b6d9d640 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ba430 tx=0x7fe2b00ee3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d5571bcf3e86070 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe29ffff640 1 -- 172.21.15.78:0/2812306934 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00b9d50 con 0x55ac96a89cb0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe29ffff640 1 -- 172.21.15.78:0/2812306934 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b003ce50 con 0x55ac96a89cb0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.994+0000 7fe29ffff640 1 -- 172.21.15.78:0/2812306934 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00b9f20 con 0x55ac96a89cb0 2024-09-16T01:17:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 msgr2=0x55ac96aa1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ba430 tx=0x7fe2b00ee3c0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 shutdown_connections 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a89cb0 0x55ac96aa1770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7f410 0x55ac96a87570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/2812306934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a7eaf0 0x55ac96a88ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 >> 172.21.15.78:0/2812306934 conn(0x55ac96a7b060 msgr2=0x55ac96a7a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 shutdown_connections 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.995+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/2812306934 wait complete. 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.996+0000 7fe2bf7d7740 1 -- start start 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.996+0000 7fe2bf7d7740 1 Processor -- start 2024-09-16T01:17:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.996+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a91940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.996+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a91940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a91940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44790/0 (socket says 172.21.15.78:44790) 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:57504/0 (socket says 172.21.15.78:57504) 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40026/0 (socket says 172.21.15.78:40026) 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b659c640 1 -- 172.21.15.78:0/3098312351 learned_addr learned my addr 172.21.15.78:0/3098312351 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b80734a0 con 0x55ac96a89cb0 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81c6150 con 0x55ac96a7eaf0 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8073b70 con 0x55ac96a7f410 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 10 client.0 _ll_register_callbacks cb 0x7fe2a80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 msgr2=0x55ac96a91940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a91940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/3098312351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 msgr2=0x55ac96a8e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2b5d9b640 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac96a8ee50 con 0x55ac96a7f410 2024-09-16T01:17:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.997+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2b003ce50 con 0x55ac96a7f410 2024-09-16T01:17:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.998+0000 7fe2b5d9b640 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8026cc0 tx=0x7fe2a8009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f97fa272716fcd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.998+0000 7fe29dffb640 10 client.25615 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.999+0000 7fe29dffb640 1 -- 172.21.15.78:0/3098312351 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a80267c0 con 0x55ac96a7f410 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.999+0000 7fe29dffb640 1 -- 172.21.15.78:0/3098312351 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fe2a8004020 con 0x55ac96a7f410 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.999+0000 7fe29dffb640 1 -- 172.21.15.78:0/3098312351 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2a8004280 con 0x55ac96a7f410 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.999+0000 7fe2bf7d7740 10 client.25615 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:15.999+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55ac96b04c80 con 0x55ac96a7f410 2024-09-16T01:17:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe29dffb640 1 -- 172.21.15.78:0/3098312351 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe2a80036b0 con 0x55ac96a7f410 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 10 client.25615 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 10 client.25615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 20 client.25615 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96b05190 0x55ac96b07580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 4 client.25615 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.000+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac969ead70 con 0x55ac96b05190 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.001+0000 7fe2b659c640 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96b05190 0x55ac96b07580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.001+0000 7fe2b659c640 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96b05190 0x55ac96b07580 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a209cdffdb41fdc1 server_cookie=f4d55781f1769eda in_seq=0 out_seq=0 2024-09-16T01:17:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.001+0000 7fe29dffb640 10 client.25615 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:16.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe29dffb640 1 -- 172.21.15.78:0/3098312351 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac969ead70 con 0x55ac96b05190 2024-09-16T01:17:16.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe29dffb640 10 client.25615 handle_command_reply: tid=0 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 client.25615 shutdown 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 msgr2=0x55ac96a91e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8026cc0 tx=0x7fe2a8009280 comp rx=0 tx=0).stop 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 shutdown_connections 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ac96b05190 0x55ac96b07580 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ac96a89cb0 0x55ac96a8e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe29dffb640 0 client.25615 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac96a7f410 0x55ac96a91e80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 --2- 172.21.15.78:0/3098312351 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac96a7eaf0 0x55ac96a91940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.010+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 >> 172.21.15.78:0/3098312351 conn(0x55ac96a7b060 msgr2=0x55ac96a799a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:16.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 shutdown_connections 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bf7d7740 1 -- 172.21.15.78:0/3098312351 wait complete. 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bf7d7740 20 client.25615 trim_cache size 0 max 16384 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:17:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:17:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 msgr2=0x7fe2a00a5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 0x7fe2a00a5d80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8197750 tx=0x7fe2a8006bd0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 msgr2=0x7fe2b810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0005e30 tx=0x7fe2b0030d80 comp rx=0 tx=0).stop 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 shutdown_connections 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe2a00a38c0 0x7fe2a00a5d80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe2b81069e0 0x7fe2b810a830 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2b8101370 0x7fe2b819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 --2- 172.21.15.78:0/3360717678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b81009d0 0x7fe2b819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 >> 172.21.15.78:0/3360717678 conn(0x7fe2b80fc6f0 msgr2=0x7fe2b810c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 shutdown_connections 2024-09-16T01:17:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:16.011+0000 7fe2bc95f640 1 -- 172.21.15.78:0/3360717678 wait complete. 2024-09-16T01:17:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:16 smithi148 ceph-mon[35030]: pgmap v199: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 863 op/s 2024-09-16T01:17:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:16 smithi078 ceph-mon[30937]: pgmap v199: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 863 op/s 2024-09-16T01:17:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:16 smithi193 ceph-mon[35701]: pgmap v199: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 863 op/s 2024-09-16T01:17:18.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:18 smithi148 ceph-mon[35030]: pgmap v200: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 779 op/s 2024-09-16T01:17:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:18 smithi078 ceph-mon[30937]: pgmap v200: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 779 op/s 2024-09-16T01:17:19.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:18 smithi193 ceph-mon[35701]: pgmap v200: 129 pgs: 129 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 19 MiB/s wr, 779 op/s 2024-09-16T01:17:20.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:20 smithi148 ceph-mon[35030]: pgmap v201: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 16 MiB/s wr, 676 op/s 2024-09-16T01:17:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 365.0 s, 1211.2 tps, lat 83.213 ms stddev 96.780 2024-09-16T01:17:21.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:20 smithi078 ceph-mon[30937]: pgmap v201: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 16 MiB/s wr, 676 op/s 2024-09-16T01:17:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:20 smithi193 ceph-mon[35701]: pgmap v201: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 16 MiB/s wr, 676 op/s 2024-09-16T01:17:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:22 smithi148 ceph-mon[35030]: pgmap v202: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T01:17:23.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:22 smithi078 ceph-mon[30937]: pgmap v202: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T01:17:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:22 smithi193 ceph-mon[35701]: pgmap v202: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 13 MiB/s wr, 585 op/s 2024-09-16T01:17:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:24 smithi148 ceph-mon[35030]: pgmap v203: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 583 op/s 2024-09-16T01:17:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:24 smithi078 ceph-mon[30937]: pgmap v203: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 583 op/s 2024-09-16T01:17:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:24 smithi193 ceph-mon[35701]: pgmap v203: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 13 MiB/s wr, 583 op/s 2024-09-16T01:17:25.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 370.0 s, 784.2 tps, lat 127.547 ms stddev 229.921 2024-09-16T01:17:26.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:26 smithi148 ceph-mon[35030]: pgmap v204: 129 pgs: 129 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-09-16T01:17:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:26 smithi078 ceph-mon[30937]: pgmap v204: 129 pgs: 129 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-09-16T01:17:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:26 smithi193 ceph-mon[35701]: pgmap v204: 129 pgs: 129 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-09-16T01:17:28.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:28 smithi148 ceph-mon[35030]: pgmap v205: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 1.34k op/s 2024-09-16T01:17:29.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:28 smithi078 ceph-mon[30937]: pgmap v205: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 1.34k op/s 2024-09-16T01:17:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:28 smithi193 ceph-mon[35701]: pgmap v205: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 1.34k op/s 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 -- 172.21.15.78:0/1445199178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d81009b0 msgr2=0x7fb3d8100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 --2- 172.21.15.78:0/1445199178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d81009b0 0x7fb3d8100e10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009900 tx=0x7fb3cc02a410 comp rx=0 tx=0).stop 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 -- 172.21.15.78:0/1445199178 shutdown_connections 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 --2- 172.21.15.78:0/1445199178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d8101350 0x7fb3d810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 --2- 172.21.15.78:0/1445199178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d81009b0 0x7fb3d8100e10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 --2- 172.21.15.78:0/1445199178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 0x7fb3d8106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 -- 172.21.15.78:0/1445199178 >> 172.21.15.78:0/1445199178 conn(0x7fb3d8077990 msgr2=0x7fb3d80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 -- 172.21.15.78:0/1445199178 shutdown_connections 2024-09-16T01:17:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.144+0000 7fb3def9f640 1 -- 172.21.15.78:0/1445199178 wait complete. 2024-09-16T01:17:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 Processor -- start 2024-09-16T01:17:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 -- start start 2024-09-16T01:17:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d8101350 0x7fb3d8196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 0x7fb3d819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810fd00 con 0x7fb3d81009b0 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3dcd3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810fb80 con 0x7fb3d8101350 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3def9f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d810fe80 con 0x7fb3d81069c0 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3dcd3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32788/0 (socket says 172.21.15.78:32788) 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/466065438 learned_addr learned my addr 172.21.15.78:0/466065438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.145+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d8101350 0x7fb3d8196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 0x7fb3d819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/466065438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 msgr2=0x7fb3d819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 0x7fb3d819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/466065438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d8101350 msgr2=0x7fb3d8196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d8101350 0x7fb3d8196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/466065438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d819b100 con 0x7fb3d81009b0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800d950 tx=0x7fb3c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3d5ffb640 1 -- 172.21.15.78:0/466065438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8017030 con 0x7fb3d81009b0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d819b3f0 con 0x7fb3d81009b0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.146+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d81a2c80 con 0x7fb3d81009b0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.147+0000 7fb3d5ffb640 1 -- 172.21.15.78:0/466065438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c8004020 con 0x7fb3d81009b0 2024-09-16T01:17:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.147+0000 7fb3d5ffb640 1 -- 172.21.15.78:0/466065438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80047a0 con 0x7fb3d81009b0 2024-09-16T01:17:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3d5ffb640 1 -- 172.21.15.78:0/466065438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb3c8004940 con 0x7fb3d81009b0 2024-09-16T01:17:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 Processor -- start 2024-09-16T01:17:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 -- start start 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbbeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560b2db232f0 con 0x560b2dbbf410 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560b2dacb270 con 0x560b2dbbeaf0 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b2db1f2b0 con 0x560b2dbc9cb0 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3dcd3b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:52784/0 (socket says 172.21.15.78:52784) 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3d7fff640 1 -- 172.21.15.78:0/306313765 learned_addr learned my addr 172.21.15.78:0/306313765 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.149+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbbeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d5ffb640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 0x7fb3c00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 -- 172.21.15.78:0/306313765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 msgr2=0x560b2dbbeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbbeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 -- 172.21.15.78:0/306313765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 msgr2=0x560b2dbc9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 -- 172.21.15.78:0/306313765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b2daca340 con 0x560b2dbc9cb0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 0x7fb3c00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d5ffb640 1 -- 172.21.15.78:0/466065438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fb3c80cf970 con 0x7fb3d81009b0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc02a930 tx=0x7fb3cc005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cca8129f2a339eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3beffd640 1 -- 172.21.15.78:0/306313765 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc035050 con 0x560b2dbc9cb0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3beffd640 1 -- 172.21.15.78:0/306313765 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc049070 con 0x560b2dbc9cb0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.150+0000 7fb3beffd640 1 -- 172.21.15.78:0/306313765 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0039f0 con 0x560b2dbc9cb0 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/306313765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 msgr2=0x560b2dbcc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc02a930 tx=0x7fb3cc005e00 comp rx=0 tx=0).stop 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 0x7fb3c00a59a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005fd0 tx=0x7fb3d0004a40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/306313765 shutdown_connections 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbcc0a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/306313765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbbeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/306313765 >> 172.21.15.78:0/306313765 conn(0x560b2dbbb060 msgr2=0x560b2dbbb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/306313765 shutdown_connections 2024-09-16T01:17:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.151+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/306313765 wait complete. 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 -- start start 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 Processor -- start 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560b2dbcc880 con 0x560b2dbbf410 2024-09-16T01:17:29.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560b2daac290 con 0x560b2dbbeaf0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b2dbcca00 con 0x560b2dbc9cb0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32812/0 (socket says 172.21.15.78:32812) 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3dcd3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3d7fff640 1 -- 172.21.15.78:0/1158665075 learned_addr learned my addr 172.21.15.78:0/1158665075 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3e1e1c740 10 client.0 _ll_register_callbacks cb 0x560b2dadd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.152+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/1158665075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 msgr2=0x560b2dbc7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/1158665075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 msgr2=0x560b2dbc6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 -- 172.21.15.78:0/1158665075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b2d836210 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc6ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80cccd0 tx=0x7fb3c8005cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1cd8ebc35784a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c809b630 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3cc049070 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fb3c8097ed0 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb3c809b1c0 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3e1e1c740 10 client.25592 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560b2d902620 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.153+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc049070 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb3c80a4040 con 0x560b2dbbeaf0 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 10 client.25592 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 10 client.25592 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 20 client.25592 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dbd9c10 0x560b2dbdc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 4 client.25592 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b2db30a20 con 0x560b2dbd9c10 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dbd9c10 0x560b2dbdc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3b7fff640 10 client.25592 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.154+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dbd9c10 0x560b2dbdc060 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2d0cdf529c34024 server_cookie=7a82111ff85e1928 in_seq=0 out_seq=0 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3b7fff640 1 -- 172.21.15.78:0/1158665075 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560b2db30a20 con 0x560b2dbd9c10 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3b7fff640 10 client.25592 handle_command_reply: tid=0 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 client.25592 shutdown 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 msgr2=0x560b2dbc6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc6ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80cccd0 tx=0x7fb3c8005cd0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 shutdown_connections 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dbd9c10 0x560b2dbdc060 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbc6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3b7fff640 0 client.25592 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/1158665075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc6ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 >> 172.21.15.78:0/1158665075 conn(0x560b2dbbb060 msgr2=0x560b2dbc0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.155+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 shutdown_connections 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.156+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/1158665075 wait complete. 2024-09-16T01:17:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.156+0000 7fb3e1e1c740 20 client.25592 trim_cache size 0 max 16384 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 Processor -- start 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 -- start start 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 0x560b2dbe1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4012880 con 0x560b2dbc9cb0 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4010150 con 0x560b2dbbeaf0 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c400dfc0 con 0x560b2dbbf410 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3dd53c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 0x560b2dbe1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3dcd3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3dd53c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32820/0 (socket says 172.21.15.78:32820) 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 0x560b2dbe1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:52814/0 (socket says 172.21.15.78:52814) 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.171+0000 7fb3dd53c640 1 -- 172.21.15.78:0/3352107347 learned_addr learned my addr 172.21.15.78:0/3352107347 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 -- 172.21.15.78:0/3352107347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 msgr2=0x560b2dbe1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 0x560b2dbe1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 -- 172.21.15.78:0/3352107347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 msgr2=0x560b2dbc7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 -- 172.21.15.78:0/3352107347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b2dbbae90 con 0x560b2dbc9cb0 2024-09-16T01:17:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0018570 tx=0x7fb3d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23686b8f2b39ebf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3be7fc640 1 -- 172.21.15.78:0/3352107347 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0028070 con 0x560b2dbc9cb0 2024-09-16T01:17:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3be7fc640 1 -- 172.21.15.78:0/3352107347 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3d001c030 con 0x560b2dbc9cb0 2024-09-16T01:17:29.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.172+0000 7fb3be7fc640 1 -- 172.21.15.78:0/3352107347 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0017b70 con 0x560b2dbc9cb0 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/3352107347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 msgr2=0x560b2dbe1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0018570 tx=0x7fb3d0017000 comp rx=0 tx=0).stop 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/3352107347 shutdown_connections 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbc9cb0 0x560b2dbe1e70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbf410 0x560b2dbe1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/3352107347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbbeaf0 0x560b2dbc7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/3352107347 >> 172.21.15.78:0/3352107347 conn(0x560b2dbbb060 msgr2=0x560b2dbc5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/3352107347 shutdown_connections 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/3352107347 wait complete. 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.173+0000 7fb3e1e1c740 1 -- start start 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 Processor -- start 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 0x560b2dbd1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c400e610 con 0x560b2dbbf410 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c400fa80 con 0x560b2dbc9cb0 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c400ece0 con 0x560b2dbbeaf0 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3e1e1c740 10 client.0 _ll_register_callbacks cb 0x7fb3c80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32826/0 (socket says 172.21.15.78:32826) 2024-09-16T01:17:29.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3d7fff640 1 -- 172.21.15.78:0/872350446 learned_addr learned my addr 172.21.15.78:0/872350446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3dd53c640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 0x560b2dbd1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.174+0000 7fb3d7fff640 1 -- 172.21.15.78:0/872350446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 msgr2=0x560b2dbd1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 0x560b2dbd1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3d7fff640 1 -- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbc9cb0 msgr2=0x560b2dbc2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3d7fff640 1 -- 172.21.15.78:0/872350446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b2dbd28f0 con 0x560b2dbbf410 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 0x560b2dbd1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3d7fff640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009900 tx=0x7fb3cc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971b39241c1e56f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc035040 con 0x560b2dbbf410 2024-09-16T01:17:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7fb3cc003c20 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3d001c030 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc003e20 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3e1e1c740 10 client.16179 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.175+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560b2dc43ed0 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3d001c030 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb3cc03aaf0 con 0x560b2dbbf410 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 10 client.16179 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 10 client.16179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 20 client.16179 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dc442d0 0x560b2dc466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 4 client.16179 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:17:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b2db2ad70 con 0x560b2dc442d0 2024-09-16T01:17:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.176+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dc442d0 0x560b2dc466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.177+0000 7fb3b7fff640 10 client.16179 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.177+0000 7fb3dcd3b640 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dc442d0 0x560b2dc466c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af9e7eb9deac9119 server_cookie=47fe8c7991781811 in_seq=0 out_seq=0 2024-09-16T01:17:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.183+0000 7fb3b7fff640 1 -- 172.21.15.78:0/872350446 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b2db2ad70 con 0x560b2dc442d0 2024-09-16T01:17:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.183+0000 7fb3b7fff640 10 client.16179 handle_command_reply: tid=0 2024-09-16T01:17:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.183+0000 7fb3e1e1c740 1 client.16179 shutdown 2024-09-16T01:17:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.183+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 msgr2=0x560b2dbd2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.183+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009900 tx=0x7fb3cc02a410 comp rx=0 tx=0).stop 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 shutdown_connections 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b2dc442d0 0x560b2dc466c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b2dbc9cb0 0x560b2dbc2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b2dbbf410 0x560b2dbd2230 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3b7fff640 0 client.16179 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 --2- 172.21.15.78:0/872350446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b2dbbeaf0 0x560b2dbd1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 >> 172.21.15.78:0/872350446 conn(0x560b2dbbb060 msgr2=0x560b2dbc05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 shutdown_connections 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 1 -- 172.21.15.78:0/872350446 wait complete. 2024-09-16T01:17:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3e1e1c740 20 client.16179 trim_cache size 0 max 16384 2024-09-16T01:17:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:17:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:17:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:17:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 msgr2=0x7fb3c00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 0x7fb3c00a59a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005fd0 tx=0x7fb3d0004a40 comp rx=0 tx=0).stop 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 msgr2=0x7fb3d8196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800d950 tx=0x7fb3c800de10 comp rx=0 tx=0).stop 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.184+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 shutdown_connections 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb3c00a34e0 0x7fb3c00a59a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3d81069c0 0x7fb3d819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d8101350 0x7fb3d8196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 --2- 172.21.15.78:0/466065438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3d81009b0 0x7fb3d8196110 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 >> 172.21.15.78:0/466065438 conn(0x7fb3d8077990 msgr2=0x7fb3d8103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 shutdown_connections 2024-09-16T01:17:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:29.185+0000 7fb3def9f640 1 -- 172.21.15.78:0/466065438 wait complete. 2024-09-16T01:17:30.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:30 smithi148 ceph-mon[35030]: pgmap v206: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-09-16T01:17:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 375.0 s, 896.8 tps, lat 111.031 ms stddev 191.545 2024-09-16T01:17:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:30 smithi078 ceph-mon[30937]: pgmap v206: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-09-16T01:17:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:30 smithi193 ceph-mon[35701]: pgmap v206: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-09-16T01:17:32.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:32 smithi148 ceph-mon[35030]: pgmap v207: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-09-16T01:17:33.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:32 smithi078 ceph-mon[30937]: pgmap v207: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-09-16T01:17:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:32 smithi193 ceph-mon[35701]: pgmap v207: 129 pgs: 129 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-09-16T01:17:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:34 smithi078 ceph-mon[30937]: pgmap v208: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-09-16T01:17:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:34 smithi193 ceph-mon[35701]: pgmap v208: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-09-16T01:17:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:34 smithi148 ceph-mon[35030]: pgmap v208: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 1.50k op/s 2024-09-16T01:17:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 380.0 s, 1177.0 tps, lat 85.336 ms stddev 114.427 2024-09-16T01:17:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:36 smithi078 ceph-mon[30937]: pgmap v209: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 21 MiB/s wr, 1.63k op/s 2024-09-16T01:17:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:36 smithi193 ceph-mon[35701]: pgmap v209: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 21 MiB/s wr, 1.63k op/s 2024-09-16T01:17:37.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:36 smithi148 ceph-mon[35030]: pgmap v209: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 21 MiB/s wr, 1.63k op/s 2024-09-16T01:17:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:38 smithi078 ceph-mon[30937]: pgmap v210: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-09-16T01:17:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:38 smithi193 ceph-mon[35701]: pgmap v210: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-09-16T01:17:39.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:38 smithi148 ceph-mon[35030]: pgmap v210: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-09-16T01:17:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 385.0 s, 1139.6 tps, lat 86.755 ms stddev 112.682 2024-09-16T01:17:41.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:40 smithi078 ceph-mon[30937]: pgmap v211: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-09-16T01:17:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:40 smithi193 ceph-mon[35701]: pgmap v211: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-09-16T01:17:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:40 smithi148 ceph-mon[35030]: pgmap v211: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-09-16T01:17:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.329+0000 7f03d943c640 1 -- 172.21.15.78:0/1789244726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4105ec0 msgr2=0x7f03d4106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.329+0000 7f03d943c640 1 --2- 172.21.15.78:0/1789244726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4105ec0 0x7f03d4106370 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009930 tx=0x7f03c402a4f0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 -- 172.21.15.78:0/1789244726 shutdown_connections 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 --2- 172.21.15.78:0/1789244726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4105ec0 0x7f03d4106370 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 --2- 172.21.15.78:0/1789244726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d41013b0 0x7f03d4105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 --2- 172.21.15.78:0/1789244726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d4100a90 0x7f03d4100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 -- 172.21.15.78:0/1789244726 >> 172.21.15.78:0/1789244726 conn(0x7f03d40fc6f0 msgr2=0x7f03d40feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.330+0000 7f03d943c640 1 -- 172.21.15.78:0/1789244726 shutdown_connections 2024-09-16T01:17:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.335+0000 7f03d943c640 1 -- 172.21.15.78:0/1789244726 wait complete. 2024-09-16T01:17:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.335+0000 7f03d943c640 1 Processor -- start 2024-09-16T01:17:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.336+0000 7f03d943c640 1 -- start start 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.339+0000 7f03d943c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d943c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 0x7f03d41a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d943c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d4105ec0 0x7f03d419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58206/0 (socket says 172.21.15.78:58206) 2024-09-16T01:17:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d2ffd640 1 -- 172.21.15.78:0/2719943403 learned_addr learned my addr 172.21.15.78:0/2719943403 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4118620 con 0x7f03d4105ec0 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41184a0 con 0x7f03d41013b0 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41187a0 con 0x7f03d4100a90 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.340+0000 7f03d27fc640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 0x7f03d41a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 -- 172.21.15.78:0/2719943403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 msgr2=0x7f03d41a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 -- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d4105ec0 msgr2=0x7f03d419e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d4105ec0 0x7f03d419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d419f030 con 0x7f03d41013b0 2024-09-16T01:17:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d27fc640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 0x7f03d41a4c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bd70 tx=0x7f03cc00be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.341+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:17:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.343+0000 7f03bbfff640 1 -- 172.21.15.78:0/2719943403 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc019070 con 0x7f03d41013b0 2024-09-16T01:17:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.343+0000 7f03bbfff640 1 -- 172.21.15.78:0/2719943403 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03cc014920 con 0x7f03d41013b0 2024-09-16T01:17:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.344+0000 7f03bbfff640 1 -- 172.21.15.78:0/2719943403 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc012950 con 0x7f03d41013b0 2024-09-16T01:17:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.344+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d419f320 con 0x7f03d41013b0 2024-09-16T01:17:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.344+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d41ab5c0 con 0x7f03d41013b0 2024-09-16T01:17:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.347+0000 7f03dc1d6740 1 Processor -- start 2024-09-16T01:17:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 -- start start 2024-09-16T01:17:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d09b0 0x55f5043d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043eccc0 0x55f5043ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f5043352f0 con 0x55f5043d09b0 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f5042dd270 con 0x55f5043d12d0 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f5043312b0 con 0x55f5043eccc0 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03d37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d09b0 0x55f5043d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33618/0 (socket says 172.21.15.78:33618) 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.348+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3607042160 learned_addr learned my addr 172.21.15.78:0/3607042160 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3607042160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043eccc0 msgr2=0x55f5043ef0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:17:42.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043eccc0 0x55f5043ef0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3607042160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d09b0 msgr2=0x55f5043d0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d09b0 0x55f5043d0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3607042160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5042dc340 con 0x55f5043d12d0 2024-09-16T01:17:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.349+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f03c800aa70 tx=0x7f03c800af30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dabd5b6a20d7a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.350+0000 7f03b97fa640 1 -- 172.21.15.78:0/3607042160 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8015070 con 0x55f5043d12d0 2024-09-16T01:17:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.350+0000 7f03b97fa640 1 -- 172.21.15.78:0/3607042160 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c800ccc0 con 0x55f5043d12d0 2024-09-16T01:17:42.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.350+0000 7f03b97fa640 1 -- 172.21.15.78:0/3607042160 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8018780 con 0x55f5043d12d0 2024-09-16T01:17:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.353+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3607042160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 msgr2=0x55f5043ec780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.353+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f03c800aa70 tx=0x7f03c800af30 comp rx=0 tx=0).stop 2024-09-16T01:17:42.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.353+0000 7f03bbfff640 1 -- 172.21.15.78:0/2719943403 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f03cc014440 con 0x7f03d41013b0 2024-09-16T01:17:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.359+0000 7f03bbfff640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 0x7f03bc0a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.359+0000 7f03d37fe640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 0x7f03bc0a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.359+0000 7f03d37fe640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 0x7f03bc0a59d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03c40057c0 tx=0x7f03c40056d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.360+0000 7f03bbfff640 1 -- 172.21.15.78:0/2719943403 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f03cc0cf850 con 0x7f03d41013b0 2024-09-16T01:17:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.361+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3607042160 shutdown_connections 2024-09-16T01:17:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.361+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043eccc0 0x55f5043ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.361+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043ec780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.361+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3607042160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d09b0 0x55f5043d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.361+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3607042160 >> 172.21.15.78:0/3607042160 conn(0x55f5043cd060 msgr2=0x55f5043cd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.363+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3607042160 shutdown_connections 2024-09-16T01:17:42.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.365+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3607042160 wait complete. 2024-09-16T01:17:42.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.365+0000 7f03dc1d6740 1 -- start start 2024-09-16T01:17:42.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.366+0000 7f03dc1d6740 1 Processor -- start 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.366+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.366+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.366+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043d93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.366+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f5043d3950 con 0x55f5043eccc0 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f5042be290 con 0x55f5043d12d0 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58218/0 (socket says 172.21.15.78:58218) 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 -- 172.21.15.78:0/833617592 learned_addr learned my addr 172.21.15.78:0/833617592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f5043d3ad0 con 0x55f5043d09b0 2024-09-16T01:17:42.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d27fc640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03dc1d6740 10 client.0 _ll_register_callbacks cb 0x55f5042ef140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 -- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 msgr2=0x55f5043de5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043de5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 -- 172.21.15.78:0/833617592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 msgr2=0x55f5043d93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043d93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 -- 172.21.15.78:0/833617592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f504048210 con 0x55f5043d09b0 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.367+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03c80183a0 tx=0x7f03c8011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21cc7f436a3cb855 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.368+0000 7f03b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.368+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03c800ccc0 con 0x55f5043d09b0 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.368+0000 7f03b6ff5640 1 -- 172.21.15.78:0/833617592 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8020070 con 0x55f5043d09b0 2024-09-16T01:17:42.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.368+0000 7f03b6ff5640 1 -- 172.21.15.78:0/833617592 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f03c8018710 con 0x55f5043d09b0 2024-09-16T01:17:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.369+0000 7f03b6ff5640 1 -- 172.21.15.78:0/833617592 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f03c8015040 con 0x55f5043d09b0 2024-09-16T01:17:42.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.371+0000 7f03dc1d6740 10 client.25639 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:42.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.371+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55f504114620 con 0x55f5043d09b0 2024-09-16T01:17:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03b6ff5640 1 -- 172.21.15.78:0/833617592 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f03c8018910 con 0x55f5043d09b0 2024-09-16T01:17:42.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 10 client.25639 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 20 client.25639 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f5043e9170 0x55f5043eb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 4 client.25639 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.372+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f03c800ccc0 con 0x55f5043e9170 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.373+0000 7f03d27fc640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f5043e9170 0x55f5043eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.373+0000 7f03d27fc640 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f5043e9170 0x55f5043eb630 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c558689849175eda server_cookie=bfd6c72bdc03e8b0 in_seq=0 out_seq=0 2024-09-16T01:17:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.373+0000 7f03b6ff5640 10 client.25639 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.374+0000 7f03b6ff5640 1 -- 172.21.15.78:0/833617592 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f03c800ccc0 con 0x55f5043e9170 2024-09-16T01:17:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.374+0000 7f03b6ff5640 10 client.25639 handle_command_reply: tid=0 2024-09-16T01:17:42.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.374+0000 7f03dc1d6740 1 client.25639 shutdown 2024-09-16T01:17:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.378+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 msgr2=0x55f5043de080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.378+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f03c80183a0 tx=0x7f03c8011a50 comp rx=0 tx=0).stop 2024-09-16T01:17:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 shutdown_connections 2024-09-16T01:17:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f5043e9170 0x55f5043eb630 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043d93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d12d0 0x55f5043de5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/833617592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043de080 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 >> 172.21.15.78:0/833617592 conn(0x55f5043cd060 msgr2=0x55f5043d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03b6ff5640 0 client.25639 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 shutdown_connections 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.382+0000 7f03dc1d6740 1 -- 172.21.15.78:0/833617592 wait complete. 2024-09-16T01:17:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.383+0000 7f03dc1d6740 20 client.25639 trim_cache size 0 max 16384 2024-09-16T01:17:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.407+0000 7f03dc1d6740 1 Processor -- start 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.407+0000 7f03dc1d6740 1 -- start start 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 0x55f5043f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41aee40 con 0x55f5043d12d0 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41af510 con 0x55f5043eccc0 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03dc1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41acde0 con 0x55f5043d09b0 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03d27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03d27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53854/0 (socket says 172.21.15.78:53854) 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03d27fc640 1 -- 172.21.15.78:0/3905530074 learned_addr learned my addr 172.21.15.78:0/3905530074 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03d37fe640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 0x55f5043f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 -- 172.21.15.78:0/3905530074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 msgr2=0x55f5043d2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.408+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043d2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 -- 172.21.15.78:0/3905530074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 msgr2=0x55f5043f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 0x55f5043f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 -- 172.21.15.78:0/3905530074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5043d10c0 con 0x55f5043d12d0 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d37fe640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 0x55f5043f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.409+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bd40 tx=0x7f03cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410b54fc19c4ce29 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.410+0000 7f03b8ff9640 1 -- 172.21.15.78:0/3905530074 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0a4460 con 0x55f5043d12d0 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.410+0000 7f03b8ff9640 1 -- 172.21.15.78:0/3905530074 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03cc014730 con 0x55f5043d12d0 2024-09-16T01:17:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.410+0000 7f03b8ff9640 1 -- 172.21.15.78:0/3905530074 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0a4230 con 0x55f5043d12d0 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3905530074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 msgr2=0x55f5043f4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bd40 tx=0x7f03cc004a40 comp rx=0 tx=0).stop 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3905530074 shutdown_connections 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043eccc0 0x55f5043f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043d12d0 0x55f5043f4680 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3905530074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d09b0 0x55f5043d2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3905530074 >> 172.21.15.78:0/3905530074 conn(0x55f5043cd060 msgr2=0x55f5043d24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3905530074 shutdown_connections 2024-09-16T01:17:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.411+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3905530074 wait complete. 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.412+0000 7f03dc1d6740 1 -- start start 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.412+0000 7f03dc1d6740 1 Processor -- start 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.412+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 0x55f5043df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03dc1d6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33636/0 (socket says 172.21.15.78:33636) 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043e3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53860/0 (socket says 172.21.15.78:53860) 2024-09-16T01:17:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3502221625 learned_addr learned my addr 172.21.15.78:0/3502221625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4073f40 con 0x55f5043eccc0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41ac710 con 0x55f5043d09b0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03d41ab970 con 0x55f5043d12d0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 0x55f5043df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3502221625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 msgr2=0x55f5043df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 0x55f5043df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3502221625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 msgr2=0x55f5043e3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043e3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d2ffd640 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5043e0090 con 0x55f5043d09b0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.413+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 0x55f5043df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03dc1d6740 10 client.0 _ll_register_callbacks cb 0x7f03c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03cc014730 con 0x55f5043d09b0 2024-09-16T01:17:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03b6ff5640 10 client.25613 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:17:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03d2ffd640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f03c800dd80 tx=0x7f03c8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f21e04e1594ccd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03b6ff5640 1 -- 172.21.15.78:0/3502221625 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8033020 con 0x55f5043d09b0 2024-09-16T01:17:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03b6ff5640 1 -- 172.21.15.78:0/3502221625 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f03c8004040 con 0x55f5043d09b0 2024-09-16T01:17:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03b6ff5640 1 -- 172.21.15.78:0/3502221625 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f03c80042a0 con 0x55f5043d09b0 2024-09-16T01:17:42.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03dc1d6740 10 client.25613 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.414+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55f5044558c0 con 0x55f5043d09b0 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03b6ff5640 1 -- 172.21.15.78:0/3502221625 <== mon.2 v2:172.21.15.148:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f03c8003690 con 0x55f5043d09b0 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 10 client.25613 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 20 client.25613 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f504455f90 0x55f504458430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 4 client.25613 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:17:42.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f504342a20 con 0x55f504455f90 2024-09-16T01:17:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f504455f90 0x55f504458430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03b6ff5640 10 client.25613 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.415+0000 7f03d27fc640 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f504455f90 0x55f504458430 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=326d8914a0cfd473 server_cookie=9f27752eb2615b62 in_seq=0 out_seq=0 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03b6ff5640 1 -- 172.21.15.78:0/3502221625 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f504342a20 con 0x55f504455f90 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03b6ff5640 10 client.25613 handle_command_reply: tid=0 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 client.25613 shutdown 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 msgr2=0x55f5043df2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f03c800dd80 tx=0x7f03c8008040 comp rx=0 tx=0).stop 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 shutdown_connections 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f504455f90 0x55f504458430 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f5043eccc0 0x55f5043e3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03b6ff5640 0 client.25613 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f5043d12d0 0x55f5043df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 --2- 172.21.15.78:0/3502221625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f5043d09b0 0x55f5043df2c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 >> 172.21.15.78:0/3502221625 conn(0x55f5043cd060 msgr2=0x55f5043cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 shutdown_connections 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 1 -- 172.21.15.78:0/3502221625 wait complete. 2024-09-16T01:17:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.422+0000 7f03dc1d6740 20 client.25613 trim_cache size 0 max 16384 2024-09-16T01:17:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:17:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:17:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:17:42.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:17:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 msgr2=0x7f03bc0a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 0x7f03bc0a59d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03c40057c0 tx=0x7f03c40056d0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 msgr2=0x7f03d41a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:42.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 0x7f03d41a4c40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bd70 tx=0x7f03cc00be60 comp rx=0 tx=0).stop 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 shutdown_connections 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f03bc0a3510 0x7f03bc0a59d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d4105ec0 0x7f03d419e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03d41013b0 0x7f03d41a4c40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 --2- 172.21.15.78:0/2719943403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d4100a90 0x7f03d41a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 >> 172.21.15.78:0/2719943403 conn(0x7f03d40fc6f0 msgr2=0x7f03d4103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 shutdown_connections 2024-09-16T01:17:42.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:42.424+0000 7f03d943c640 1 -- 172.21.15.78:0/2719943403 wait complete. 2024-09-16T01:17:43.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:42 smithi078 ceph-mon[30937]: pgmap v212: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.6 MiB/s wr, 851 op/s 2024-09-16T01:17:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:42 smithi193 ceph-mon[35701]: pgmap v212: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.6 MiB/s wr, 851 op/s 2024-09-16T01:17:43.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:42 smithi148 ceph-mon[35030]: pgmap v212: 129 pgs: 129 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.6 MiB/s wr, 851 op/s 2024-09-16T01:17:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:44 smithi078 ceph-mon[30937]: pgmap v213: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 7.6 MiB/s wr, 633 op/s 2024-09-16T01:17:45.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:44 smithi193 ceph-mon[35701]: pgmap v213: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 7.6 MiB/s wr, 633 op/s 2024-09-16T01:17:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:44 smithi148 ceph-mon[35030]: pgmap v213: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 7.6 MiB/s wr, 633 op/s 2024-09-16T01:17:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 390.0 s, 840.0 tps, lat 116.771 ms stddev 164.011 2024-09-16T01:17:47.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:46 smithi078 ceph-mon[30937]: pgmap v214: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-09-16T01:17:47.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:46 smithi193 ceph-mon[35701]: pgmap v214: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-09-16T01:17:47.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:46 smithi148 ceph-mon[35030]: pgmap v214: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-09-16T01:17:49.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:48 smithi078 ceph-mon[30937]: pgmap v215: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-09-16T01:17:49.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:48 smithi193 ceph-mon[35701]: pgmap v215: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-09-16T01:17:49.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:48 smithi148 ceph-mon[35030]: pgmap v215: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-09-16T01:17:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 395.0 s, 915.2 tps, lat 111.559 ms stddev 143.296 2024-09-16T01:17:51.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:50 smithi078 ceph-mon[30937]: pgmap v216: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-09-16T01:17:51.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:50 smithi193 ceph-mon[35701]: pgmap v216: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-09-16T01:17:51.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:50 smithi148 ceph-mon[35030]: pgmap v216: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-09-16T01:17:53.039 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:52 smithi078 ceph-mon[30937]: pgmap v217: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-09-16T01:17:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:52 smithi193 ceph-mon[35701]: pgmap v217: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-09-16T01:17:53.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:52 smithi148 ceph-mon[35030]: pgmap v217: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-09-16T01:17:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:54 smithi078 ceph-mon[30937]: pgmap v218: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.6 MiB/s wr, 929 op/s 2024-09-16T01:17:55.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:54 smithi193 ceph-mon[35701]: pgmap v218: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.6 MiB/s wr, 929 op/s 2024-09-16T01:17:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:54 smithi148 ceph-mon[35030]: pgmap v218: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 9.6 MiB/s wr, 929 op/s 2024-09-16T01:17:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 400.0 s, 730.0 tps, lat 128.459 ms stddev 163.068 2024-09-16T01:17:56.936 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:56 smithi148 ceph-mon[35030]: pgmap v219: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-09-16T01:17:56.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:56 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:17:57.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:56 smithi078 ceph-mon[30937]: pgmap v219: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-09-16T01:17:57.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:56 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:17:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:56 smithi193 ceph-mon[35701]: pgmap v219: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-09-16T01:17:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:56 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.470+0000 7f836b277640 1 -- 172.21.15.78:0/1436094394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8364102ce0 msgr2=0x7f83641030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.470+0000 7f836b277640 1 --2- 172.21.15.78:0/1436094394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8364102ce0 0x7f83641030c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8360009900 tx=0x7f836002a480 comp rx=0 tx=0).stop 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- 172.21.15.78:0/1436094394 shutdown_connections 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- 172.21.15.78:0/1436094394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f836410c0b0 0x7f836410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- 172.21.15.78:0/1436094394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f836410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- 172.21.15.78:0/1436094394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8364102ce0 0x7f83641030c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- 172.21.15.78:0/1436094394 >> 172.21.15.78:0/1436094394 conn(0x7f8364077990 msgr2=0x7f83640fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- 172.21.15.78:0/1436094394 shutdown_connections 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- 172.21.15.78:0/1436094394 wait complete. 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 Processor -- start 2024-09-16T01:17:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- start start 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8364102ce0 0x7f8364198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f8364198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8364111ef0 con 0x7f836410c0b0 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8364111d70 con 0x7f8364103600 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.471+0000 7f836b277640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8364112070 con 0x7f8364102ce0 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53340/0 (socket says 172.21.15.78:53340) 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8369814640 1 -- 172.21.15.78:0/2129474299 learned_addr learned my addr 172.21.15.78:0/2129474299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f8364198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8369013640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8364102ce0 0x7f8364198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 -- 172.21.15.78:0/2129474299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8364102ce0 msgr2=0x7f8364198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8364102ce0 0x7f8364198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 -- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 msgr2=0x7f836419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 -- 172.21.15.78:0/2129474299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836419d410 con 0x7f8364103600 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8369814640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.472+0000 7f8368812640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f8364198960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f835400c970 tx=0x7f835400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.473+0000 7f835a7fc640 1 -- 172.21.15.78:0/2129474299 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8354007bb0 con 0x7f8364103600 2024-09-16T01:17:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.473+0000 7f835a7fc640 1 -- 172.21.15.78:0/2129474299 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8354007d50 con 0x7f8364103600 2024-09-16T01:17:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.474+0000 7f835a7fc640 1 -- 172.21.15.78:0/2129474299 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8354007bb0 con 0x7f8364103600 2024-09-16T01:17:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.479+0000 7f836b277640 1 -- 172.21.15.78:0/2129474299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f836419d700 con 0x7f8364103600 2024-09-16T01:17:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.479+0000 7f836b277640 1 -- 172.21.15.78:0/2129474299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83641a4f90 con 0x7f8364103600 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.482+0000 7f836e010740 1 Processor -- start 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.482+0000 7f836e010740 1 -- start start 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370ccd0 0x55a5f370d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a5f36712f0 con 0x55a5f370ccd0 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a5f3619270 con 0x55a5f370d5f0 2024-09-16T01:17:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.483+0000 7f836e010740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a5f366d2b0 con 0x55a5f3728fe0 2024-09-16T01:17:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.485+0000 7f8369013640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.485+0000 7f8368812640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.485+0000 7f8368812640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:48144/0 (socket says 172.21.15.78:48144) 2024-09-16T01:17:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.485+0000 7f8368812640 1 -- 172.21.15.78:0/3612911966 learned_addr learned my addr 172.21.15.78:0/3612911966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8368812640 1 -- 172.21.15.78:0/3612911966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 msgr2=0x55a5f3728aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8368812640 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8368812640 1 -- 172.21.15.78:0/3612911966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370ccd0 msgr2=0x55a5f370d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8368812640 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370ccd0 0x55a5f370d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8368812640 1 -- 172.21.15.78:0/3612911966 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5f3618340 con 0x55a5f3728fe0 2024-09-16T01:17:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.486+0000 7f8369013640 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:17:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.487+0000 7f835a7fc640 1 -- 172.21.15.78:0/2129474299 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f835400f450 con 0x7f8364103600 2024-09-16T01:17:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.487+0000 7f8368812640 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f83540cc280 tx=0x7f83540d7570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7f5b2b747b5fbf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.488+0000 7f834effd640 1 -- 172.21.15.78:0/3612911966 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83540e2070 con 0x55a5f3728fe0 2024-09-16T01:17:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.488+0000 7f834effd640 1 -- 172.21.15.78:0/3612911966 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8354007d50 con 0x55a5f3728fe0 2024-09-16T01:17:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.488+0000 7f835a7fc640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 0x7f83500a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f835a7fc640 1 -- 172.21.15.78:0/2129474299 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f83540079b0 con 0x7f8364103600 2024-09-16T01:17:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f8369814640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 0x7f83500a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f834effd640 1 -- 172.21.15.78:0/3612911966 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83540dc970 con 0x55a5f3728fe0 2024-09-16T01:17:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 -- 172.21.15.78:0/3612911966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 msgr2=0x55a5f372b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f83540cc280 tx=0x7f83540d7570 comp rx=0 tx=0).stop 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f8369814640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 0x7f83500a5db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f835c00c4b0 tx=0x7f835c00b040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 -- 172.21.15.78:0/3612911966 shutdown_connections 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f372b3d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 --2- 172.21.15.78:0/3612911966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370ccd0 0x55a5f370d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 -- 172.21.15.78:0/3612911966 >> 172.21.15.78:0/3612911966 conn(0x55a5f3709380 msgr2=0x55a5f3709760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 -- 172.21.15.78:0/3612911966 shutdown_connections 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.489+0000 7f836e010740 1 -- 172.21.15.78:0/3612911966 wait complete. 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.490+0000 7f836e010740 1 -- start start 2024-09-16T01:17:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.490+0000 7f836e010740 1 Processor -- start 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.490+0000 7f836e010740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.490+0000 7f836e010740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370d5f0 0x55a5f3725cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:37232/0 (socket says 172.21.15.78:37232) 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f836e010740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f3726440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f836e010740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a5f372bb70 con 0x55a5f370d5f0 2024-09-16T01:17:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f836e010740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a5f35fa290 con 0x55a5f370ccd0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f836e010740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a5f372bcf0 con 0x55a5f3728fe0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 -- 172.21.15.78:0/2237754806 learned_addr learned my addr 172.21.15.78:0/2237754806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f836e010740 10 client.0 _ll_register_callbacks cb 0x55a5f362b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8368812640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370d5f0 0x55a5f3725cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369814640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f3726440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 -- 172.21.15.78:0/2237754806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 msgr2=0x55a5f3726440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f3726440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 -- 172.21.15.78:0/2237754806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370d5f0 msgr2=0x55a5f3725cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370d5f0 0x55a5f3725cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 -- 172.21.15.78:0/2237754806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5f3384210 con 0x55a5f370ccd0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369814640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f3726440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f8369013640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8360005ab0 tx=0x7f83600317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f578c097bf911b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.491+0000 7f834cff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836003f070 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f8360004450 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8354007d50 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8360035070 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f836e010740 10 client.25619 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a5f3450620 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83600319d0 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.492+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f83600035f0 con 0x55a5f370ccd0 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 10 client.25619 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 10 client.25619 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 20 client.25619 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f3739050 0x55a5f373b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 4 client.25619 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:17:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8354007d50 con 0x55a5f3739050 2024-09-16T01:17:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f8368812640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f3739050 0x55a5f373b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f8368812640 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f3739050 0x55a5f373b440 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a563000b79008d server_cookie=3445e4ade039dd37 in_seq=0 out_seq=0 2024-09-16T01:17:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.493+0000 7f834cff9640 10 client.25619 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f834cff9640 1 -- 172.21.15.78:0/2237754806 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f8354007d50 con 0x55a5f3739050 2024-09-16T01:17:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f834cff9640 10 client.25619 handle_command_reply: tid=0 2024-09-16T01:17:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 client.25619 shutdown 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 msgr2=0x55a5f3728a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8360005ab0 tx=0x7f83600317e0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 shutdown_connections 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f3739050 0x55a5f373b440 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f3728fe0 0x55a5f3726440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f370d5f0 0x55a5f3725cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 --2- 172.21.15.78:0/2237754806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370ccd0 0x55a5f3728a10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 >> 172.21.15.78:0/2237754806 conn(0x55a5f3709380 msgr2=0x55a5f3708640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f834cff9640 0 client.25619 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.494+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 shutdown_connections 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.495+0000 7f836e010740 1 -- 172.21.15.78:0/2237754806 wait complete. 2024-09-16T01:17:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.495+0000 7f836e010740 20 client.25619 trim_cache size 0 max 16384 2024-09-16T01:17:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.514+0000 7f836e010740 1 Processor -- start 2024-09-16T01:17:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.514+0000 7f836e010740 1 -- start start 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3728520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f83641a9200 con 0x55a5f3728fe0 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83641a6ad0 con 0x55a5f370d5f0 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f836e010740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8364073c00 con 0x55a5f370ccd0 2024-09-16T01:17:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53380/0 (socket says 172.21.15.78:53380) 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 -- 172.21.15.78:0/3966475834 learned_addr learned my addr 172.21.15.78:0/3966475834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8368812640 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369013640 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3728520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 -- 172.21.15.78:0/3966475834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 msgr2=0x55a5f3728520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3728520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 -- 172.21.15.78:0/3966475834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 msgr2=0x55a5f3728a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.515+0000 7f8369814640 1 -- 172.21.15.78:0/3966475834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5f37267b0 con 0x55a5f3728fe0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f8369814640 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f835c0157f0 tx=0x7f835c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e66f45b98ce27d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f834e7fc640 1 -- 172.21.15.78:0/3966475834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c01b040 con 0x55a5f3728fe0 2024-09-16T01:17:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f834e7fc640 1 -- 172.21.15.78:0/3966475834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835c026070 con 0x55a5f3728fe0 2024-09-16T01:17:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f834e7fc640 1 -- 172.21.15.78:0/3966475834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c021e50 con 0x55a5f3728fe0 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 -- 172.21.15.78:0/3966475834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 msgr2=0x55a5f3740d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f835c0157f0 tx=0x7f835c011000 comp rx=0 tx=0).stop 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 -- 172.21.15.78:0/3966475834 shutdown_connections 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f3740d50 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3728a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 --2- 172.21.15.78:0/3966475834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3728520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.516+0000 7f836e010740 1 -- 172.21.15.78:0/3966475834 >> 172.21.15.78:0/3966475834 conn(0x55a5f3709380 msgr2=0x55a5f3708640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- 172.21.15.78:0/3966475834 shutdown_connections 2024-09-16T01:17:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- 172.21.15.78:0/3966475834 wait complete. 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- start start 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 Processor -- start 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3730fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3731510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8364074250 con 0x55a5f3728fe0 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83641a6400 con 0x55a5f370d5f0 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83641a5660 con 0x55a5f370ccd0 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.517+0000 7f836e010740 10 client.0 _ll_register_callbacks cb 0x7f8360009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53394/0 (socket says 172.21.15.78:53394) 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 -- 172.21.15.78:0/2661045505 learned_addr learned my addr 172.21.15.78:0/2661045505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:17:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8368812640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3731510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369013640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3730fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 -- 172.21.15.78:0/2661045505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 msgr2=0x55a5f3730fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3730fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 -- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 msgr2=0x55a5f3731510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3731510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 -- 172.21.15.78:0/2661045505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5f379f8e0 con 0x55a5f3728fe0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369013640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3730fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8369814640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f835c00cd40 tx=0x7f835c021720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed9aed92f66ed34 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8343fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c031070 con 0x55a5f3728fe0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9525+0+0 (secure 0 0 0) 0x7f835c013b50 con 0x55a5f3728fe0 2024-09-16T01:17:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f835c026070 con 0x55a5f3728fe0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f835c03d050 con 0x55a5f3728fe0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f836e010740 10 client.16203 fetch_fsmap learned FSMap version 31 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.518+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a5f37a8510 con 0x55a5f3728fe0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835c026070 con 0x55a5f3728fe0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f835c029380 con 0x55a5f3728fe0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 10 client.16203 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 20 client.16203 populate_metadata read hostname 'smithi078' 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f37a8ba0 0x55a5f37aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 4 client.16203 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5f367ea20 con 0x55a5f37a8ba0 2024-09-16T01:17:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f8369013640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f37a8ba0 0x55a5f37aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:17:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f8369013640 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f37a8ba0 0x55a5f37aafc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240334b90ce890b7 server_cookie=c57cda5779d53edb in_seq=0 out_seq=0 2024-09-16T01:17:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.519+0000 7f8343fff640 10 client.16203 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f8343fff640 1 -- 172.21.15.78:0/2661045505 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5f367ea20 con 0x55a5f37a8ba0 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f8343fff640 10 client.16203 handle_command_reply: tid=0 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 client.16203 shutdown 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 msgr2=0x55a5f379e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f835c00cd40 tx=0x7f835c021720 comp rx=0 tx=0).stop 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 shutdown_connections 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a5f37a8ba0 0x55a5f37aafc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a5f3728fe0 0x55a5f379e340 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f8343fff640 0 client.16203 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a5f370d5f0 0x55a5f3731510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 --2- 172.21.15.78:0/2661045505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5f370ccd0 0x55a5f3730fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 >> 172.21.15.78:0/2661045505 conn(0x55a5f3709380 msgr2=0x55a5f370e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.640+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 shutdown_connections 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f836e010740 1 -- 172.21.15.78:0/2661045505 wait complete. 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f836e010740 20 client.16203 trim_cache size 0 max 16384 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:17:57.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 msgr2=0x7f83500a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 0x7f83500a5db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f835c00c4b0 tx=0x7f835c00b040 comp rx=0 tx=0).stop 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 msgr2=0x7f8364198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f8364198960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f835400c970 tx=0x7f835400ce30 comp rx=0 tx=0).stop 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 shutdown_connections 2024-09-16T01:17:57.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f83500a38f0 0x7f83500a5db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f836410c0b0 0x7f836419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8364103600 0x7f8364198960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 --2- 172.21.15.78:0/2129474299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8364102ce0 0x7f8364198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 >> 172.21.15.78:0/2129474299 conn(0x7f8364077990 msgr2=0x7f83640fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 shutdown_connections 2024-09-16T01:17:57.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:17:57.641+0000 7f834ffff640 1 -- 172.21.15.78:0/2129474299 wait complete. 2024-09-16T01:17:59.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:58 smithi078 ceph-mon[30937]: pgmap v220: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 MiB/s wr, 839 op/s 2024-09-16T01:17:59.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:58 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:17:59.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:58 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:17:59.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:17:58 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:17:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:58 smithi193 ceph-mon[35701]: pgmap v220: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 MiB/s wr, 839 op/s 2024-09-16T01:17:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:58 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:17:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:58 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:17:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:17:58 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:17:59.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:58 smithi148 ceph-mon[35030]: pgmap v220: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 MiB/s wr, 839 op/s 2024-09-16T01:17:59.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:58 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:17:59.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:58 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:17:59.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:17:58 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:18:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 405.0 s, 643.4 tps, lat 166.038 ms stddev 270.817 2024-09-16T01:18:01.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:00 smithi078 ceph-mon[30937]: pgmap v221: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 839 op/s 2024-09-16T01:18:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:00 smithi193 ceph-mon[35701]: pgmap v221: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 839 op/s 2024-09-16T01:18:01.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:00 smithi148 ceph-mon[35030]: pgmap v221: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 839 op/s 2024-09-16T01:18:03.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:02 smithi078 ceph-mon[30937]: pgmap v222: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 818 op/s 2024-09-16T01:18:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:02 smithi193 ceph-mon[35701]: pgmap v222: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 818 op/s 2024-09-16T01:18:03.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:02 smithi148 ceph-mon[35030]: pgmap v222: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.9 MiB/s wr, 818 op/s 2024-09-16T01:18:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:04 smithi078 ceph-mon[30937]: pgmap v223: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.6 MiB/s wr, 684 op/s 2024-09-16T01:18:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:04 smithi193 ceph-mon[35701]: pgmap v223: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.6 MiB/s wr, 684 op/s 2024-09-16T01:18:05.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:04 smithi148 ceph-mon[35030]: pgmap v223: 129 pgs: 129 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.6 MiB/s wr, 684 op/s 2024-09-16T01:18:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 410.0 s, 933.6 tps, lat 105.566 ms stddev 138.288 2024-09-16T01:18:07.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:06 smithi078 ceph-mon[30937]: pgmap v224: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.5 MiB/s wr, 882 op/s 2024-09-16T01:18:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:06 smithi193 ceph-mon[35701]: pgmap v224: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.5 MiB/s wr, 882 op/s 2024-09-16T01:18:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:06 smithi148 ceph-mon[35030]: pgmap v224: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.5 MiB/s wr, 882 op/s 2024-09-16T01:18:09.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:08 smithi078 ceph-mon[30937]: pgmap v225: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-09-16T01:18:09.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:08 smithi193 ceph-mon[35701]: pgmap v225: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-09-16T01:18:09.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:08 smithi148 ceph-mon[35030]: pgmap v225: 129 pgs: 129 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-09-16T01:18:09.709 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.7 2024-09-16T01:18:09.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:18:10.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:09 smithi078 ceph-mon[30937]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T01:18:10.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:09 smithi193 ceph-mon[35701]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T01:18:10.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:09 smithi148 ceph-mon[35030]: osdmap e97: 12 total, 12 up, 12 in 2024-09-16T01:18:10.241 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:18:10.905 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:10 smithi078 ceph-mon[30937]: pgmap v227: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 727 op/s 2024-09-16T01:18:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.906+0000 7f350c3c0640 1 -- 172.21.15.78:0/2482105659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 msgr2=0x7f3504112580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.906+0000 7f350c3c0640 1 --2- 172.21.15.78:0/2482105659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f3504112580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ca0 tx=0x7f34fc02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.908+0000 7f350c3c0640 1 -- 172.21.15.78:0/2482105659 shutdown_connections 2024-09-16T01:18:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.908+0000 7f350c3c0640 1 --2- 172.21.15.78:0/2482105659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f3504114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.908+0000 7f350c3c0640 1 --2- 172.21.15.78:0/2482105659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f3504112580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.908+0000 7f350c3c0640 1 --2- 172.21.15.78:0/2482105659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.908+0000 7f350c3c0640 1 -- 172.21.15.78:0/2482105659 >> 172.21.15.78:0/2482105659 conn(0x7f3504075e20 msgr2=0x7f3504076230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:10.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.909+0000 7f350c3c0640 1 -- 172.21.15.78:0/2482105659 shutdown_connections 2024-09-16T01:18:10.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.909+0000 7f350c3c0640 1 -- 172.21.15.78:0/2482105659 wait complete. 2024-09-16T01:18:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.910+0000 7f350c3c0640 1 Processor -- start 2024-09-16T01:18:10.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.910+0000 7f350c3c0640 1 -- start start 2024-09-16T01:18:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504112060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f350410b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3504118520 con 0x7f3504112ac0 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35041183a0 con 0x7f35041070d0 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350c3c0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35041186a0 con 0x7f35041067b0 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350a95d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350a95d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45334/0 (socket says 172.21.15.78:45334) 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350a95d640 1 -- 172.21.15.78:0/3987298269 learned_addr learned my addr 172.21.15.78:0/3987298269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:10.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350995b640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f350410b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.911+0000 7f350a15c640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 -- 172.21.15.78:0/3987298269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 msgr2=0x7f3504112060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504112060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 -- 172.21.15.78:0/3987298269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 msgr2=0x7f350410b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f350410b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 -- 172.21.15.78:0/3987298269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350410bc80 con 0x7f3504112ac0 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a15c640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:18:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350995b640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f350410b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350a95d640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3500009fd0 tx=0x7f350000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f34fb7fe640 1 -- 172.21.15.78:0/3987298269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350000cd80 con 0x7f3504112ac0 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f34fb7fe640 1 -- 172.21.15.78:0/3987298269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35000035f0 con 0x7f3504112ac0 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350410bf70 con 0x7f3504112ac0 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f34fb7fe640 1 -- 172.21.15.78:0/3987298269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500010670 con 0x7f3504112ac0 2024-09-16T01:18:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.912+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350410fe80 con 0x7f3504112ac0 2024-09-16T01:18:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.916+0000 7f34fb7fe640 1 -- 172.21.15.78:0/3987298269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3500010810 con 0x7f3504112ac0 2024-09-16T01:18:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.917+0000 7f34fb7fe640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 0x7f34f00a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.917+0000 7f34fb7fe640 1 -- 172.21.15.78:0/3987298269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7f35000d14f0 con 0x7f3504112ac0 2024-09-16T01:18:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.917+0000 7f350a15c640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 0x7f34f00a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.929+0000 7f21278dc640 1 -- 172.21.15.78:0/3594386609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010cf30 msgr2=0x7f2120119410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.929+0000 7f21278dc640 1 --2- 172.21.15.78:0/3594386609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010cf30 0x7f2120119410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2118009cd0 tx=0x7f211802a8e0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 -- 172.21.15.78:0/3594386609 shutdown_connections 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 --2- 172.21.15.78:0/3594386609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010cf30 0x7f2120119410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 --2- 172.21.15.78:0/3594386609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f212010c590 0x7f212010c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 --2- 172.21.15.78:0/3594386609 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2120112640 0x7f2120112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 -- 172.21.15.78:0/3594386609 >> 172.21.15.78:0/3594386609 conn(0x7f212006cdc0 msgr2=0x7f212006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:10.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 -- 172.21.15.78:0/3594386609 shutdown_connections 2024-09-16T01:18:10.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.930+0000 7f21278dc640 1 -- 172.21.15.78:0/3594386609 wait complete. 2024-09-16T01:18:10.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.931+0000 7f21278dc640 1 Processor -- start 2024-09-16T01:18:10.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.931+0000 7f21278dc640 1 -- start start 2024-09-16T01:18:10.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.931+0000 7f21278dc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.931+0000 7f21278dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 0x7f2120116300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.931+0000 7f21278dc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 0x7f2120116840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f21278dc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f212011c9a0 con 0x7f2120112640 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f21278dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f212011c820 con 0x7f212010cf30 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f21278dc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f212011cb20 con 0x7f212010c590 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125e79640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 0x7f2120116840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36926/0 (socket says 172.21.15.78:36926) 2024-09-16T01:18:10.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 -- 172.21.15.78:0/3741164213 learned_addr learned my addr 172.21.15.78:0/3741164213 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2124e77640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 0x7f2120116300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 -- 172.21.15.78:0/3741164213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 msgr2=0x7f2120116300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 0x7f2120116300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 -- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 msgr2=0x7f2120116840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 0x7f2120116840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 -- 172.21.15.78:0/3741164213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21201170b0 con 0x7f212010c590 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125e79640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 0x7f2120116840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2124e77640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 0x7f2120116300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.932+0000 7f2125678640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f211c00e9f0 tx=0x7f211c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.933+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c00cd80 con 0x7f212010c590 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.933+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21201a8f20 con 0x7f212010c590 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.933+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201a9480 con 0x7f212010c590 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.933+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f211c010910 con 0x7f212010c590 2024-09-16T01:18:10.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.933+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0188f0 con 0x7f212010c590 2024-09-16T01:18:10.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 415.0 s, 708.6 tps, lat 142.145 ms stddev 194.764 2024-09-16T01:18:10.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.934+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21040052c0 con 0x7f212010c590 2024-09-16T01:18:10.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.935+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f211c018bb0 con 0x7f212010c590 2024-09-16T01:18:10.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.936+0000 7f21167fc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 0x7f210c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.936+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7f211c0d0390 con 0x7f212010c590 2024-09-16T01:18:10.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.936+0000 7f2124e77640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 0x7f210c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.937+0000 7f2124e77640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 0x7f210c0a58f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2120117ac0 tx=0x7f2110005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.941+0000 7f350a15c640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 0x7f34f00a5970 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34f40058c0 tx=0x7f34f4005850 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:10.941+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f211c09c930 con 0x7f212010c590 2024-09-16T01:18:10.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 Processor -- start 2024-09-16T01:18:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 -- start start 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792f410 0x55b037939770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037939cb0 0x55b03793c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b0378932f0 con 0x55b03792eaf0 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b03783b270 con 0x55b03792f410 2024-09-16T01:18:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350f22f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b03788f2b0 con 0x55b037939cb0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350a15c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350a15c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45360/0 (socket says 172.21.15.78:45360) 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350a15c640 1 -- 172.21.15.78:0/990646770 learned_addr learned my addr 172.21.15.78:0/990646770 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350a95d640 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037939cb0 0x55b03793c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.943+0000 7f350995b640 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792f410 0x55b037939770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 -- 172.21.15.78:0/990646770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037939cb0 msgr2=0x55b03793c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037939cb0 0x55b03793c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 -- 172.21.15.78:0/990646770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792f410 msgr2=0x55b037939770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792f410 0x55b037939770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 -- 172.21.15.78:0/990646770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03783a340 con 0x55b03792eaf0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f350a15c640 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f34f4010010 tx=0x7f34f4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328ceb9db94eacd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f34f8bfe640 1 -- 172.21.15.78:0/990646770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4022070 con 0x55b03792eaf0 2024-09-16T01:18:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f34f8bfe640 1 -- 172.21.15.78:0/990646770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f401eb30 con 0x55b03792eaf0 2024-09-16T01:18:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.944+0000 7f34f8bfe640 1 -- 172.21.15.78:0/990646770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4010b10 con 0x55b03792eaf0 2024-09-16T01:18:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 -- 172.21.15.78:0/990646770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 msgr2=0x55b03792eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f34f4010010 tx=0x7f34f4012000 comp rx=0 tx=0).stop 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 -- 172.21.15.78:0/990646770 shutdown_connections 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037939cb0 0x55b03793c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792f410 0x55b037939770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 --2- 172.21.15.78:0/990646770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792eaf0 0x55b03792eed0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 -- 172.21.15.78:0/990646770 >> 172.21.15.78:0/990646770 conn(0x55b03792b060 msgr2=0x55b03792b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 -- 172.21.15.78:0/990646770 shutdown_connections 2024-09-16T01:18:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.945+0000 7f350f22f740 1 -- 172.21.15.78:0/990646770 wait complete. 2024-09-16T01:18:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350f22f740 1 -- start start 2024-09-16T01:18:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350f22f740 1 Processor -- start 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350f22f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 0x55b037936bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350f22f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792f410 0x55b037937110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350f22f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 0x55b037937650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.946+0000 7f350a15c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 0x55b037936bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a15c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 0x55b037936bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36946/0 (socket says 172.21.15.78:36946) 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a15c640 1 -- 172.21.15.78:0/678837489 learned_addr learned my addr 172.21.15.78:0/678837489 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b03793c880 con 0x55b03792f410 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b03781c290 con 0x55b037939cb0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b03793ca00 con 0x55b03792eaf0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 0x55b037937650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350995b640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792f410 0x55b037937110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350f22f740 10 client.0 _ll_register_callbacks cb 0x55b03784d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 -- 172.21.15.78:0/678837489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 msgr2=0x55b037936bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 0x55b037936bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 -- 172.21.15.78:0/678837489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792f410 msgr2=0x55b037937110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792f410 0x55b037937110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0375a6210 con 0x55b037939cb0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f401eb30 con 0x55b037939cb0 2024-09-16T01:18:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.947+0000 7f350a95d640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 0x55b037937650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35000ce7a0 tx=0x7f350009cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fd2816ecb168f90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f34edffb640 10 client.25640 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f34edffb640 1 -- 172.21.15.78:0/678837489 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35000b6060 con 0x55b037939cb0 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f34edffb640 1 -- 172.21.15.78:0/678837489 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7f35000999a0 con 0x55b037939cb0 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f34edffb640 1 -- 172.21.15.78:0/678837489 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f350009d330 con 0x55b037939cb0 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f350f22f740 10 client.25640 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:10.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.948+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55b037672620 con 0x55b037939cb0 2024-09-16T01:18:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f34edffb640 1 -- 172.21.15.78:0/678837489 <== mon.2 v2:172.21.15.148:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f35000037a0 con 0x55b037939cb0 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 10 client.25640 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 10 client.25640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 20 client.25640 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379499b0 0x55b03794be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 4 client.25640 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.949+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0378a0a20 con 0x55b0379499b0 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.950+0000 7f350a15c640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379499b0 0x55b03794be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.950+0000 7f34edffb640 10 client.25640 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.950+0000 7f350a15c640 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379499b0 0x55b03794be50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=738902c182058f30 server_cookie=a6e3c558cbd60d75 in_seq=0 out_seq=0 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.951+0000 7f34edffb640 1 -- 172.21.15.78:0/678837489 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b0378a0a20 con 0x55b0379499b0 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.951+0000 7f34edffb640 10 client.25640 handle_command_reply: tid=0 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.951+0000 7f350f22f740 1 client.25640 shutdown 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 msgr2=0x55b037937650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 0x55b037937650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35000ce7a0 tx=0x7f350009cd50 comp rx=0 tx=0).stop 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 shutdown_connections 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379499b0 0x55b03794be50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b037939cb0 0x55b037937650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f34edffb640 0 client.25640 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b03792f410 0x55b037937110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 --2- 172.21.15.78:0/678837489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792eaf0 0x55b037936bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 >> 172.21.15.78:0/678837489 conn(0x55b03792b060 msgr2=0x55b037930dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 shutdown_connections 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 1 -- 172.21.15.78:0/678837489 wait complete. 2024-09-16T01:18:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.952+0000 7f350f22f740 20 client.25640 trim_cache size 0 max 16384 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 Processor -- start 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 -- start start 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b0379315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b037931f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35041b8020 con 0x55b037939cb0 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35041b58f0 con 0x55b03792eaf0 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350f22f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3504069d90 con 0x55b03792f410 2024-09-16T01:18:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350a95d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350a95d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45376/0 (socket says 172.21.15.78:45376) 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.978+0000 7f350a95d640 1 -- 172.21.15.78:0/557986032 learned_addr learned my addr 172.21.15.78:0/557986032 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 -- 172.21.15.78:0/557986032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 msgr2=0x55b037931f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b037931f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 -- 172.21.15.78:0/557986032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 msgr2=0x55b0379315b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b0379315b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 -- 172.21.15.78:0/557986032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03792e920 con 0x55b037939cb0 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f350a95d640 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f350009d4f0 tx=0x7f35000bd050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faeaaea69f4bf9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f34effff640 1 -- 172.21.15.78:0/557986032 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35000a16e0 con 0x55b037939cb0 2024-09-16T01:18:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.979+0000 7f34effff640 1 -- 172.21.15.78:0/557986032 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f401eb30 con 0x55b037939cb0 2024-09-16T01:18:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.980+0000 7f350f22f740 1 -- 172.21.15.78:0/557986032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 msgr2=0x55b037937480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.980+0000 7f350f22f740 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f350009d4f0 tx=0x7f35000bd050 comp rx=0 tx=0).stop 2024-09-16T01:18:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.986+0000 7f350f22f740 1 -- 172.21.15.78:0/557986032 shutdown_connections 2024-09-16T01:18:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.986+0000 7f350f22f740 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b037937480 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.986+0000 7f350f22f740 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b037931f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.986+0000 7f350f22f740 1 --2- 172.21.15.78:0/557986032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b0379315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.986+0000 7f350f22f740 1 -- 172.21.15.78:0/557986032 >> 172.21.15.78:0/557986032 conn(0x55b03792b060 msgr2=0x55b037938c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.995+0000 7f350f22f740 1 -- 172.21.15.78:0/557986032 shutdown_connections 2024-09-16T01:18:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.996+0000 7f350f22f740 1 -- 172.21.15.78:0/557986032 wait complete. 2024-09-16T01:18:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 -- start start 2024-09-16T01:18:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 Processor -- start 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b03793db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b03793e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3504110970 con 0x55b037939cb0 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35041b5220 con 0x55b03792eaf0 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3504073e70 con 0x55b03792f410 2024-09-16T01:18:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:10.997+0000 7f350f22f740 10 client.0 _ll_register_callbacks cb 0x7f35000058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36962/0 (socket says 172.21.15.78:36962) 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 -- 172.21.15.78:0/2053115034 learned_addr learned my addr 172.21.15.78:0/2053115034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 -- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 msgr2=0x55b03793db20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b03793db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 -- 172.21.15.78:0/2053115034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 msgr2=0x55b03793e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b03793e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.001+0000 7f350995b640 1 -- 172.21.15.78:0/2053115034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0379b38e0 con 0x55b03792f410 2024-09-16T01:18:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.002+0000 7f350995b640 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ca0 tx=0x7f34fc005f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324d2d348871a592 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.002+0000 7f34edffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:18:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.007+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc031860 con 0x55b03792f410 2024-09-16T01:18:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.007+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7f34fc004040 con 0x55b03792f410 2024-09-16T01:18:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.008+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f401eb30 con 0x55b03792f410 2024-09-16T01:18:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.012+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34fc0098c0 con 0x55b03792f410 2024-09-16T01:18:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.012+0000 7f350f22f740 10 client.25675 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.012+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b0379b4c00 con 0x55b03792f410 2024-09-16T01:18:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.017+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f401eb30 con 0x55b03792f410 2024-09-16T01:18:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.017+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f34fc02fbe0 con 0x55b03792f410 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 10 client.25675 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 10 client.25675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 20 client.25675 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379b5380 0x55b0379b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 4 client.25675 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:18:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.019+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b03789ad70 con 0x55b0379b5380 2024-09-16T01:18:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.021+0000 7f350a15c640 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379b5380 0x55b0379b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:11.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.023+0000 7f350a15c640 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379b5380 0x55b0379b77a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce6794172abbf7db server_cookie=42e6406da987f35f in_seq=0 out_seq=0 2024-09-16T01:18:11.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.024+0000 7f34edffb640 10 client.25675 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.044+0000 7f34edffb640 1 -- 172.21.15.78:0/2053115034 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b03789ad70 con 0x55b0379b5380 2024-09-16T01:18:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.044+0000 7f34edffb640 10 client.25675 handle_command_reply: tid=0 2024-09-16T01:18:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.044+0000 7f350f22f740 1 client.25675 shutdown 2024-09-16T01:18:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:10 smithi193 ceph-mon[35701]: pgmap v227: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 727 op/s 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 msgr2=0x55b03793e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ca0 tx=0x7f34fc005f50 comp rx=0 tx=0).stop 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 shutdown_connections 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b0379b5380 0x55b0379b77a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b037939cb0 0x55b03793e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b03792f410 0x55b03793e060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 --2- 172.21.15.78:0/2053115034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b03792eaf0 0x55b03793db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 >> 172.21.15.78:0/2053115034 conn(0x55b03792b060 msgr2=0x55b0379299a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f34edffb640 0 client.25675 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 shutdown_connections 2024-09-16T01:18:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.045+0000 7f350f22f740 1 -- 172.21.15.78:0/2053115034 wait complete. 2024-09-16T01:18:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.047+0000 7f350f22f740 20 client.25675 trim_cache size 0 max 16384 2024-09-16T01:18:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:18:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:18:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:18:11.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 msgr2=0x7f34f00a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 0x7f34f00a5970 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34f40058c0 tx=0x7f34f4005850 comp rx=0 tx=0).stop 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 msgr2=0x7f350410b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f3500009fd0 tx=0x7f350000edd0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 shutdown_connections 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f34f00a34b0 0x7f34f00a5970 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3504112ac0 0x7f350410b6c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f35041070d0 0x7f350410b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 --2- 172.21.15.78:0/3987298269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35041067b0 0x7f3504112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.048+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 >> 172.21.15.78:0/3987298269 conn(0x7f3504075e20 msgr2=0x7f3504105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.049+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 shutdown_connections 2024-09-16T01:18:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:11.049+0000 7f350c3c0640 1 -- 172.21.15.78:0/3987298269 wait complete. 2024-09-16T01:18:11.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:10 smithi148 ceph-mon[35030]: pgmap v227: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 727 op/s 2024-09-16T01:18:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:18:11.324 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:18:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:18:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.321+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2104005880 con 0x7f212010c590 2024-09-16T01:18:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.322+0000 7f21167fc640 1 -- 172.21.15.78:0/3741164213 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f211c09c230 con 0x7f212010c590 2024-09-16T01:18:11.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.325+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 msgr2=0x7f210c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 0x7f210c0a58f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2120117ac0 tx=0x7f2110005f90 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 msgr2=0x7f2120119250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f211c00e9f0 tx=0x7f211c00eeb0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 shutdown_connections 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f210c0a3430 0x7f210c0a58f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2120112640 0x7f2120116840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f212010cf30 0x7f2120116300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 --2- 172.21.15.78:0/3741164213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f212010c590 0x7f2120119250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 >> 172.21.15.78:0/3741164213 conn(0x7f212006cdc0 msgr2=0x7f212010ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 shutdown_connections 2024-09-16T01:18:11.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:11.326+0000 7f21278dc640 1 -- 172.21.15.78:0/3741164213 wait complete. 2024-09-16T01:18:12.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:11 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3741164213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:12.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:11 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3741164213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:12.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:11 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3741164213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:12.366 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0' 2024-09-16T01:18:12.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0 2024-09-16T01:18:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:18:13.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:12 smithi078 ceph-mon[30937]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 6.7 MiB/s wr, 697 op/s 2024-09-16T01:18:13.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:12 smithi193 ceph-mon[35701]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 6.7 MiB/s wr, 697 op/s 2024-09-16T01:18:13.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:12 smithi148 ceph-mon[35030]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 6.7 MiB/s wr, 697 op/s 2024-09-16T01:18:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.440+0000 7fd66709d640 1 -- 172.21.15.78:0/3312125240 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660101de0 msgr2=0x7fd6601021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.440+0000 7fd66709d640 1 --2- 172.21.15.78:0/3312125240 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660101de0 0x7fd6601021c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd654009900 tx=0x7fd65402a410 comp rx=0 tx=0).stop 2024-09-16T01:18:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 -- 172.21.15.78:0/3312125240 shutdown_connections 2024-09-16T01:18:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 --2- 172.21.15.78:0/3312125240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 0x7fd66010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 --2- 172.21.15.78:0/3312125240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660102790 0x7fd6601067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 --2- 172.21.15.78:0/3312125240 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660101de0 0x7fd6601021c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 -- 172.21.15.78:0/3312125240 >> 172.21.15.78:0/3312125240 conn(0x7fd6600fd8e0 msgr2=0x7fd6600ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 -- 172.21.15.78:0/3312125240 shutdown_connections 2024-09-16T01:18:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 -- 172.21.15.78:0/3312125240 wait complete. 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 Processor -- start 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.441+0000 7fd66709d640 1 -- start start 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 0x7fd660070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 0x7fd6600713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd660111fb0 con 0x7fd660101de0 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd664e39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd664e39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45404/0 (socket says 172.21.15.78:45404) 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd664e39640 1 -- 172.21.15.78:0/1308442701 learned_addr learned my addr 172.21.15.78:0/1308442701 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 0x7fd660070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd660111e30 con 0x7fd660102790 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd660112130 con 0x7fd660106f10 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd66563a640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 0x7fd6600713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 -- 172.21.15.78:0/1308442701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 msgr2=0x7fd6600713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 0x7fd6600713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 -- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 msgr2=0x7fd660075e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd65ffff640 1 -- 172.21.15.78:0/1308442701 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd660071b70 con 0x7fd660102790 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.442+0000 7fd664e39640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd65ffff640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 0x7fd660070eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00d6d0 tx=0x7fd64c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd65dffb640 1 -- 172.21.15.78:0/1308442701 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c017070 con 0x7fd660102790 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6601af4d0 con 0x7fd660102790 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd65dffb640 1 -- 172.21.15.78:0/1308442701 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c009820 con 0x7fd660102790 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd65dffb640 1 -- 172.21.15.78:0/1308442701 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c004cd0 con 0x7fd660102790 2024-09-16T01:18:13.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.443+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6601afa00 con 0x7fd660102790 2024-09-16T01:18:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.445+0000 7fd65dffb640 1 -- 172.21.15.78:0/1308442701 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fd64c0099d0 con 0x7fd660102790 2024-09-16T01:18:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 Processor -- start 2024-09-16T01:18:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 -- start start 2024-09-16T01:18:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbdeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbebd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e21db442a0 con 0x55e21dbdf080 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e21daebb10 con 0x55e21dbde760 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd669476740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e21db40260 con 0x55e21dbe9940 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd664e39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd664e39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45414/0 (socket says 172.21.15.78:45414) 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd664e39640 1 -- 172.21.15.78:0/3692831187 learned_addr learned my addr 172.21.15.78:0/3692831187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd65ffff640 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd65dffb640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 0x7fd6500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.446+0000 7fd66563a640 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbdeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd66563a640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 0x7fd6500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 -- 172.21.15.78:0/3692831187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 msgr2=0x55e21dbebd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbebd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 -- 172.21.15.78:0/3692831187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 msgr2=0x55e21dbdeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbdeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 -- 172.21.15.78:0/3692831187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21d8b5b90 con 0x55e21dbdf080 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd65dffb640 1 -- 172.21.15.78:0/1308442701 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7fd64c0cfcb0 con 0x7fd660102790 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd664e39640 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6540096c0 tx=0x7fd6540316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd8de609a7e587f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd64affd640 1 -- 172.21.15.78:0/3692831187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654040070 con 0x55e21dbdf080 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd64affd640 1 -- 172.21.15.78:0/3692831187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654005040 con 0x55e21dbdf080 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.447+0000 7fd64affd640 1 -- 172.21.15.78:0/3692831187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654035040 con 0x55e21dbdf080 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- 172.21.15.78:0/3692831187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 msgr2=0x55e21dbe9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd66563a640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 0x7fd6500a5870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd658009fd0 tx=0x7fd65800d040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6540096c0 tx=0x7fd6540316e0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- 172.21.15.78:0/3692831187 shutdown_connections 2024-09-16T01:18:13.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbebd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbe9400 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 --2- 172.21.15.78:0/3692831187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbdeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- 172.21.15.78:0/3692831187 >> 172.21.15.78:0/3692831187 conn(0x55e21dbdacf0 msgr2=0x55e21dbdb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- 172.21.15.78:0/3692831187 shutdown_connections 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- 172.21.15.78:0/3692831187 wait complete. 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 -- start start 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.448+0000 7fd669476740 1 Processor -- start 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbe6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e21dbec560 con 0x55e21dbe9940 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e21dacc6e0 con 0x55e21dbde760 2024-09-16T01:18:13.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e21dbec6e0 con 0x55e21dbdf080 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd669476740 10 client.0 _ll_register_callbacks cb 0x55e21daf0a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45416/0 (socket says 172.21.15.78:45416) 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 -- 172.21.15.78:0/1713129598 learned_addr learned my addr 172.21.15.78:0/1713129598 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd664e39640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd65ffff640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbe6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 -- 172.21.15.78:0/1713129598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 msgr2=0x55e21dbe6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbe6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 -- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 msgr2=0x55e21dbe67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.449+0000 7fd66563a640 1 -- 172.21.15.78:0/1713129598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e21dbf59d0 con 0x55e21dbe9940 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd66563a640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd658018c10 tx=0x7fd658004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bff2c528b725be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd63ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658014040 con 0x55e21dbe9940 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7fd658022610 con 0x55e21dbe9940 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd654005040 con 0x55e21dbe9940 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd658030050 con 0x55e21dbe9940 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd669476740 10 client.16236 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e21dbf8360 con 0x55e21dbe9940 2024-09-16T01:18:13.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.450+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654005040 con 0x55e21dbe9940 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.451+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd65802b7d0 con 0x55e21dbe9940 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 10 client.16236 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 10 client.16236 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 20 client.16236 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dbf9870 0x55e21dbfbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 4 client.16236 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:13.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.452+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e21db519d0 con 0x55e21dbf9870 2024-09-16T01:18:13.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.463+0000 7fd664e39640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dbf9870 0x55e21dbfbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.483+0000 7fd63ffff640 10 client.16236 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.483+0000 7fd664e39640 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dbf9870 0x55e21dbfbd10 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15ba3820128cf130 server_cookie=3ab8b85cffbda16a in_seq=0 out_seq=0 2024-09-16T01:18:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.484+0000 7fd63ffff640 1 -- 172.21.15.78:0/1713129598 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e21db519d0 con 0x55e21dbf9870 2024-09-16T01:18:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.484+0000 7fd63ffff640 10 client.16236 handle_command_reply: tid=0 2024-09-16T01:18:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.484+0000 7fd669476740 1 client.16236 shutdown 2024-09-16T01:18:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.486+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 msgr2=0x55e21dbe7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.486+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd658018c10 tx=0x7fd658004ad0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 shutdown_connections 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dbf9870 0x55e21dbfbd10 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dbe7220 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbe6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 --2- 172.21.15.78:0/1713129598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 >> 172.21.15.78:0/1713129598 conn(0x55e21dbdacf0 msgr2=0x55e21dbe0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.487+0000 7fd63ffff640 0 client.16236 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.493+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 shutdown_connections 2024-09-16T01:18:13.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.495+0000 7fd669476740 1 -- 172.21.15.78:0/1713129598 wait complete. 2024-09-16T01:18:13.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.500+0000 7fd669476740 20 client.16236 trim_cache size 0 max 16384 2024-09-16T01:18:13.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.526+0000 7fd669476740 1 Processor -- start 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.526+0000 7fd669476740 1 -- start start 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbd9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601bc530 con 0x55e21dbe9940 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601b9e00 con 0x55e21dbde760 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd669476740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601b7c70 con 0x55e21dbdf080 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbd9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45418/0 (socket says 172.21.15.78:45418) 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbd9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:37012/0 (socket says 172.21.15.78:37012) 2024-09-16T01:18:13.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 -- 172.21.15.78:0/2273920174 learned_addr learned my addr 172.21.15.78:0/2273920174 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd664e39640 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 -- 172.21.15.78:0/2273920174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 msgr2=0x55e21dbd9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbd9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 -- 172.21.15.78:0/2273920174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 msgr2=0x55e21dbe11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 -- 172.21.15.78:0/2273920174 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21dbe5750 con 0x55e21dbe9940 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.527+0000 7fd66563a640 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd658009730 tx=0x7fd6580096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1deae29357d7597a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd64a7fc640 1 -- 172.21.15.78:0/2273920174 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658018e50 con 0x55e21dbe9940 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd64a7fc640 1 -- 172.21.15.78:0/2273920174 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658027070 con 0x55e21dbe9940 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd64a7fc640 1 -- 172.21.15.78:0/2273920174 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658017610 con 0x55e21dbe9940 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd669476740 1 -- 172.21.15.78:0/2273920174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 msgr2=0x55e21dc015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd669476740 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd658009730 tx=0x7fd6580096c0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.528+0000 7fd669476740 1 -- 172.21.15.78:0/2273920174 shutdown_connections 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbe9940 0x55e21dc015e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbdf080 0x55e21dbd9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 --2- 172.21.15.78:0/2273920174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbe11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 -- 172.21.15.78:0/2273920174 >> 172.21.15.78:0/2273920174 conn(0x55e21dbdacf0 msgr2=0x55e21dbd9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 -- 172.21.15.78:0/2273920174 shutdown_connections 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 -- 172.21.15.78:0/2273920174 wait complete. 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 -- start start 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.529+0000 7fd669476740 1 Processor -- start 2024-09-16T01:18:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbf19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbedea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601b82c0 con 0x55e21dbdf080 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601b9730 con 0x55e21dbde760 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45428/0 (socket says 172.21.15.78:45428) 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd65ffff640 1 -- 172.21.15.78:0/927323733 learned_addr learned my addr 172.21.15.78:0/927323733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6601b8990 con 0x55e21dbe9940 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd664e39640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbf19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd669476740 10 client.0 _ll_register_callbacks cb 0x7fd658018860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.530+0000 7fd66563a640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbedea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 -- 172.21.15.78:0/927323733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 msgr2=0x55e21dbedea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbedea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 -- 172.21.15.78:0/927323733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 msgr2=0x55e21dbf1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 -- 172.21.15.78:0/927323733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e21dbee6a0 con 0x55e21dbde760 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd65ffff640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd664e39640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbf19f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20a68d42aa6e8183 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd63ffff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654053070 con 0x55e21dbde760 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9613+0+0 (secure 0 0 0) 0x7fd65404f5a0 con 0x55e21dbde760 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd658027070 con 0x55e21dbde760 2024-09-16T01:18:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd65405f050 con 0x55e21dbde760 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd669476740 10 client.25658 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e21dc65080 con 0x55e21dbde760 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.531+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658027070 con 0x55e21dbde760 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fd654004c90 con 0x55e21dbde760 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 10 client.25658 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 10 client.25658 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 20 client.25658 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dc65420 0x55e21dc67810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 4 client.25658 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}] 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}) -- 0x55e21db4bd20 con 0x55e21dc65420 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.532+0000 7fd65ffff640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dc65420 0x55e21dc67810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.533+0000 7fd65ffff640 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dc65420 0x55e21dc67810 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e96ee2e93bf496 server_cookie=da043d51309dde2e in_seq=0 out_seq=0 2024-09-16T01:18:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.533+0000 7fd63ffff640 10 client.25658 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd63ffff640 1 -- 172.21.15.78:0/927323733 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e21db4bd20 con 0x55e21dc65420 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd63ffff640 10 client.25658 handle_command_reply: tid=0 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 client.25658 shutdown 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 msgr2=0x55e21dbf19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbf19f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65402a410 comp rx=0 tx=0).stop 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 shutdown_connections 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e21dc65420 0x55e21dc67810 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dbe9940 0x55e21dbedea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e21dbdf080 0x55e21dbf1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd63ffff640 0 client.25658 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 --2- 172.21.15.78:0/927323733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e21dbde760 0x55e21dbf19f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 >> 172.21.15.78:0/927323733 conn(0x55e21dbdacf0 msgr2=0x55e21dbe6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 shutdown_connections 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.966+0000 7fd669476740 1 -- 172.21.15.78:0/927323733 wait complete. 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd669476740 20 client.25658 trim_cache size 0 max 16384 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 26, 2024-09-16T01:18:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "bc889bfa": { 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 26, 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:18:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 143.7, 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:13.972 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:18:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 msgr2=0x7fd6500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 0x7fd6500a5870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd658009fd0 tx=0x7fd65800d040 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 msgr2=0x7fd660070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 0x7fd660070eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00d6d0 tx=0x7fd64c00db90 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 shutdown_connections 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd6500a33b0 0x7fd6500a5870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd660106f10 0x7fd6600713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd660102790 0x7fd660070eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 --2- 172.21.15.78:0/1308442701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd660101de0 0x7fd660075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.967+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 >> 172.21.15.78:0/1308442701 conn(0x7fd6600fd8e0 msgr2=0x7fd6600fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.968+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 shutdown_connections 2024-09-16T01:18:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:13.968+0000 7fd66709d640 1 -- 172.21.15.78:0/1308442701 wait complete. 2024-09-16T01:18:14.707 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bc889bfa', quiesce took 5.0 seconds. Will release after: 18.7 2024-09-16T01:18:14.932 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:14 smithi078 ceph-mon[30937]: pgmap v229: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.2 MiB/s wr, 721 op/s 2024-09-16T01:18:15.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:14 smithi193 ceph-mon[35701]: pgmap v229: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.2 MiB/s wr, 721 op/s 2024-09-16T01:18:15.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:14 smithi148 ceph-mon[35030]: pgmap v229: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.2 MiB/s wr, 721 op/s 2024-09-16T01:18:15.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 420.0 s, 345.3 tps, lat 151.483 ms stddev 170.650 2024-09-16T01:18:17.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:16 smithi078 ceph-mon[30937]: pgmap v230: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.9 MiB/s wr, 721 op/s 2024-09-16T01:18:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:16 smithi193 ceph-mon[35701]: pgmap v230: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.9 MiB/s wr, 721 op/s 2024-09-16T01:18:17.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:16 smithi148 ceph-mon[35030]: pgmap v230: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.9 MiB/s wr, 721 op/s 2024-09-16T01:18:19.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:18 smithi078 ceph-mon[30937]: pgmap v231: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.7 MiB/s wr, 695 op/s 2024-09-16T01:18:19.057 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:18 smithi193 ceph-mon[35701]: pgmap v231: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.7 MiB/s wr, 695 op/s 2024-09-16T01:18:19.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:18 smithi148 ceph-mon[35030]: pgmap v231: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.7 MiB/s wr, 695 op/s 2024-09-16T01:18:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 425.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:18:21.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:20 smithi078 ceph-mon[30937]: pgmap v232: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 5.2 MiB/s wr, 569 op/s 2024-09-16T01:18:21.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:20 smithi078 ceph-mon[30937]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T01:18:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:20 smithi193 ceph-mon[35701]: pgmap v232: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 5.2 MiB/s wr, 569 op/s 2024-09-16T01:18:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:20 smithi193 ceph-mon[35701]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T01:18:21.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:20 smithi148 ceph-mon[35030]: pgmap v232: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 5.2 MiB/s wr, 569 op/s 2024-09-16T01:18:21.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:20 smithi148 ceph-mon[35030]: osdmap e98: 12 total, 12 up, 12 in 2024-09-16T01:18:23.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:22 smithi078 ceph-mon[30937]: pgmap v234: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 3.4 MiB/s wr, 372 op/s 2024-09-16T01:18:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:22 smithi193 ceph-mon[35701]: pgmap v234: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 3.4 MiB/s wr, 372 op/s 2024-09-16T01:18:23.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:22 smithi148 ceph-mon[35030]: pgmap v234: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 3.4 MiB/s wr, 372 op/s 2024-09-16T01:18:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 -- 172.21.15.78:0/1382630875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb8101350 msgr2=0x7f1cb810d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/1382630875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb8101350 0x7f1cb810d880 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009cd0 tx=0x7f1cb002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 -- 172.21.15.78:0/1382630875 shutdown_connections 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/1382630875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb8101350 0x7f1cb810d880 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/1382630875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb81009b0 0x7f1cb8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/1382630875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb8106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 -- 172.21.15.78:0/1382630875 >> 172.21.15.78:0/1382630875 conn(0x7f1cb8077990 msgr2=0x7f1cb80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 -- 172.21.15.78:0/1382630875 shutdown_connections 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.145+0000 7f1cbfd42640 1 -- 172.21.15.78:0/1382630875 wait complete. 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 Processor -- start 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 -- start start 2024-09-16T01:18:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb8101350 0x7f1cb8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8111ef0 con 0x7f1cb81009b0 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8111d70 con 0x7f1cb81069c0 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbfd42640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb8112070 con 0x7f1cb8101350 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52680/0 (socket says 172.21.15.78:52680) 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 -- 172.21.15.78:0/2524966060 learned_addr learned my addr 172.21.15.78:0/2524966060 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb8101350 0x7f1cb8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 -- 172.21.15.78:0/2524966060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb8101350 msgr2=0x7f1cb8198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb8101350 0x7f1cb8198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 -- 172.21.15.78:0/2524966060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 msgr2=0x7f1cb819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbdade640 1 -- 172.21.15.78:0/2524966060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb819d3b0 con 0x7f1cb81009b0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.146+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1cbdade640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400c890 tx=0x7f1cb400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1caaffd640 1 -- 172.21.15.78:0/2524966060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4004220 con 0x7f1cb81009b0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1caaffd640 1 -- 172.21.15.78:0/2524966060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cb40043c0 con 0x7f1cb81009b0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb819d6a0 con 0x7f1cb81009b0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1caaffd640 1 -- 172.21.15.78:0/2524966060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb40049e0 con 0x7f1cb81009b0 2024-09-16T01:18:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.147+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81a4f30 con 0x7f1cb81009b0 2024-09-16T01:18:24.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.149+0000 7f1caaffd640 1 -- 172.21.15.78:0/2524966060 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f1cb40079b0 con 0x7f1cb81009b0 2024-09-16T01:18:24.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.149+0000 7f1cc2ad8740 1 Processor -- start 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 -- start start 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234812f0 0x55a92348fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 0x55a9234924b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a9233e52f0 con 0x55a9234809d0 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a92338d270 con 0x55a9234812f0 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a9233e12b0 con 0x55a9234900c0 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1caaffd640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 0x7f1ca40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52682/0 (socket says 172.21.15.78:52682) 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbd2dd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 0x55a9234924b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 -- 172.21.15.78:0/2040357192 learned_addr learned my addr 172.21.15.78:0/2040357192 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:24.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 0x7f1ca40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbdade640 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234812f0 0x55a92348fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1caaffd640 1 -- 172.21.15.78:0/2524966060 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1cb40d0130 con 0x7f1cb81009b0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 -- 172.21.15.78:0/2040357192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 msgr2=0x55a9234924b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 0x55a9234924b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 -- 172.21.15.78:0/2040357192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234812f0 msgr2=0x55a92348fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234812f0 0x55a92348fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbe2df640 1 -- 172.21.15.78:0/2040357192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a92338c340 con 0x55a9234809d0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.150+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 0x55a9234924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.151+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009590 tx=0x7f1cb0031e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc3e7f610548fea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.151+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 0x7f1ca40a5920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0003970 tx=0x7f1cb0036000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.151+0000 7f1ca3fff640 1 -- 172.21.15.78:0/2040357192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb004c070 con 0x55a9234809d0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.151+0000 7f1ca3fff640 1 -- 172.21.15.78:0/2040357192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cb0036070 con 0x55a9234809d0 2024-09-16T01:18:24.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.151+0000 7f1ca3fff640 1 -- 172.21.15.78:0/2040357192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0030c00 con 0x55a9234809d0 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2040357192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 msgr2=0x55a923480db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009590 tx=0x7f1cb0031e70 comp rx=0 tx=0).stop 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2040357192 shutdown_connections 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234900c0 0x55a9234924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234812f0 0x55a92348fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2040357192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234809d0 0x55a923480db0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2040357192 >> 172.21.15.78:0/2040357192 conn(0x55a92347d060 msgr2=0x55a92347d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2040357192 shutdown_connections 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2040357192 wait complete. 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 -- start start 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 Processor -- start 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.152+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a92348d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a92348d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a923492c90 con 0x55a9234900c0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a92336e290 con 0x55a9234809d0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a923492e10 con 0x55a9234812f0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 10 client.0 _ll_register_callbacks cb 0x55a92339f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52690/0 (socket says 172.21.15.78:52690) 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 -- 172.21.15.78:0/4172351635 learned_addr learned my addr 172.21.15.78:0/4172351635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbdade640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a92348d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a92348d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 -- 172.21.15.78:0/4172351635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 msgr2=0x55a92348d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a92348d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 -- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 msgr2=0x55a92348d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a92348d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 -- 172.21.15.78:0/4172351635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9230f8210 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbdade640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a92348d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1cb002a920 tx=0x7f1cb0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b01eea68baeef74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1ca17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb004c070 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1cb002e040 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cb0036070 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1cb005d050 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 10 client.16254 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.153+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a9231c4620 con 0x55a9234900c0 2024-09-16T01:18:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cb0036070 con 0x55a9234900c0 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1cb0047070 con 0x55a9234900c0 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 10 client.16254 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 20 client.16254 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92349ff10 0x55a9234a2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 4 client.16254 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9233f2a20 con 0x55a92349ff10 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cbdade640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92349ff10 0x55a9234a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1ca17fa640 10 client.16254 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.154+0000 7f1cbdade640 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92349ff10 0x55a9234a2300 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f01b959705894b13 server_cookie=69b56a2f63c8b4e8 in_seq=0 out_seq=0 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.155+0000 7f1ca17fa640 1 -- 172.21.15.78:0/4172351635 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a9233f2a20 con 0x55a92349ff10 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.155+0000 7f1ca17fa640 10 client.16254 handle_command_reply: tid=0 2024-09-16T01:18:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.155+0000 7f1cc2ad8740 1 client.16254 shutdown 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 msgr2=0x55a92348da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1cb002a920 tx=0x7f1cb0005dc0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 shutdown_connections 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92349ff10 0x55a9234a2300 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a92348da90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a92348d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1ca17fa640 0 client.16254 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/4172351635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a92348d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 >> 172.21.15.78:0/4172351635 conn(0x55a92347d060 msgr2=0x55a923482dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 shutdown_connections 2024-09-16T01:18:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/4172351635 wait complete. 2024-09-16T01:18:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.156+0000 7f1cc2ad8740 20 client.16254 trim_cache size 0 max 16384 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 Processor -- start 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 -- start start 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234809d0 0x55a92348d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234900c0 0x55a9234a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81af7f0 con 0x55a9234812f0 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81ad0c0 con 0x55a9234900c0 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81aaf30 con 0x55a9234809d0 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52692/0 (socket says 172.21.15.78:52692) 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 -- 172.21.15.78:0/2450426680 learned_addr learned my addr 172.21.15.78:0/2450426680 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 -- 172.21.15.78:0/2450426680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234809d0 msgr2=0x55a92348d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234809d0 0x55a92348d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 -- 172.21.15.78:0/2450426680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234900c0 msgr2=0x55a9234a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.170+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234900c0 0x55a9234a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cbd2dd640 1 -- 172.21.15.78:0/2450426680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a92347ce90 con 0x55a9234812f0 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1cac004650 tx=0x7f1cac00bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd3d234c3ceb23a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1ca37fe640 1 -- 172.21.15.78:0/2450426680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac003c50 con 0x55a9234812f0 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1ca37fe640 1 -- 172.21.15.78:0/2450426680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac003df0 con 0x55a9234812f0 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1ca37fe640 1 -- 172.21.15.78:0/2450426680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac011440 con 0x55a9234812f0 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2450426680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 msgr2=0x55a9234a79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1cac004650 tx=0x7f1cac00bc90 comp rx=0 tx=0).stop 2024-09-16T01:18:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2450426680 shutdown_connections 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234900c0 0x55a9234a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234812f0 0x55a9234a79b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/2450426680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234809d0 0x55a92348d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2450426680 >> 172.21.15.78:0/2450426680 conn(0x55a92347d060 msgr2=0x55a92348f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2450426680 shutdown_connections 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.171+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/2450426680 wait complete. 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 -- start start 2024-09-16T01:18:24.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 Processor -- start 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a923494510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a923494a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81ab580 con 0x55a9234900c0 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81ac9f0 con 0x55a9234809d0 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb81abc50 con 0x55a9234812f0 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cc2ad8740 10 client.0 _ll_register_callbacks cb 0x7f1cac004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cbe2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52698/0 (socket says 172.21.15.78:52698) 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cbe2df640 1 -- 172.21.15.78:0/3843758690 learned_addr learned my addr 172.21.15.78:0/3843758690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.172+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a923494a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbdade640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a923494510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 -- 172.21.15.78:0/3843758690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 msgr2=0x55a923494a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a923494a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 -- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 msgr2=0x55a923494510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a923494510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 -- 172.21.15.78:0/3843758690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a923509ec0 con 0x55a9234900c0 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbd2dd640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a923494a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cbe2df640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009df0 tx=0x7f1cb0031530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e44ceb535ab6423 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb004c040 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f1cb0055070 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cac003df0 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1cb0069050 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 10 client.16260 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a92350b1d0 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cac003df0 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1cb0047040 con 0x55a9234900c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 10 client.16260 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 20 client.16260 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92350b8c0 0x55a92350dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 4 client.16260 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.173+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a9233ecd70 con 0x55a92350b8c0 2024-09-16T01:18:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.174+0000 7f1cbdade640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92350b8c0 0x55a92350dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.174+0000 7f1ca17fa640 10 client.16260 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.174+0000 7f1cbdade640 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92350b8c0 0x55a92350dce0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2833b38a13d2306d server_cookie=77be5c21862d8941 in_seq=0 out_seq=0 2024-09-16T01:18:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1ca17fa640 1 -- 172.21.15.78:0/3843758690 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a9233ecd70 con 0x55a92350b8c0 2024-09-16T01:18:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1ca17fa640 10 client.16260 handle_command_reply: tid=0 2024-09-16T01:18:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 client.16260 shutdown 2024-09-16T01:18:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 msgr2=0x55a923509980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009df0 tx=0x7f1cb0031530 comp rx=0 tx=0).stop 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 shutdown_connections 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a92350b8c0 0x55a92350dce0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a9234900c0 0x55a923509980 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9234812f0 0x55a923494a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1ca17fa640 0 client.16260 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 --2- 172.21.15.78:0/3843758690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a9234809d0 0x55a923494510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 >> 172.21.15.78:0/3843758690 conn(0x55a92347d060 msgr2=0x55a92347b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 shutdown_connections 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 1 -- 172.21.15.78:0/3843758690 wait complete. 2024-09-16T01:18:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.182+0000 7f1cc2ad8740 20 client.16260 trim_cache size 0 max 16384 2024-09-16T01:18:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:18:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:18:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:18:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 msgr2=0x7f1ca40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 0x7f1ca40a5920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0003970 tx=0x7f1cb0036000 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 msgr2=0x7f1cb81983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400c890 tx=0x7f1cb400cd50 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 shutdown_connections 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1ca40a3460 0x7f1ca40a5920 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cb81069c0 0x7f1cb819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cb8101350 0x7f1cb8198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 --2- 172.21.15.78:0/2524966060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cb81009b0 0x7f1cb81983c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 >> 172.21.15.78:0/2524966060 conn(0x7f1cb8077990 msgr2=0x7f1cb81037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:24.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 shutdown_connections 2024-09-16T01:18:24.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:24.183+0000 7f1cbfd42640 1 -- 172.21.15.78:0/2524966060 wait complete. 2024-09-16T01:18:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:24 smithi078 ceph-mon[30937]: pgmap v235: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-09-16T01:18:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:24 smithi193 ceph-mon[35701]: pgmap v235: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-09-16T01:18:25.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:24 smithi148 ceph-mon[35030]: pgmap v235: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-09-16T01:18:25.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 430.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:18:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:26 smithi078 ceph-mon[30937]: pgmap v236: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 581 KiB/s wr, 72 op/s 2024-09-16T01:18:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:26 smithi193 ceph-mon[35701]: pgmap v236: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 581 KiB/s wr, 72 op/s 2024-09-16T01:18:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:26 smithi148 ceph-mon[35030]: pgmap v236: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 581 KiB/s wr, 72 op/s 2024-09-16T01:18:29.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:28 smithi078 ceph-mon[30937]: pgmap v237: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 94 KiB/s wr, 13 op/s 2024-09-16T01:18:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:28 smithi193 ceph-mon[35701]: pgmap v237: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 94 KiB/s wr, 13 op/s 2024-09-16T01:18:29.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:28 smithi148 ceph-mon[35030]: pgmap v237: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 94 KiB/s wr, 13 op/s 2024-09-16T01:18:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 435.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:18:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:30 smithi078 ceph-mon[30937]: pgmap v238: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-09-16T01:18:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:30 smithi193 ceph-mon[35701]: pgmap v238: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-09-16T01:18:31.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:30 smithi148 ceph-mon[35030]: pgmap v238: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-09-16T01:18:33.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:32 smithi078 ceph-mon[30937]: pgmap v239: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 68 KiB/s wr, 9 op/s 2024-09-16T01:18:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:32 smithi193 ceph-mon[35701]: pgmap v239: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 68 KiB/s wr, 9 op/s 2024-09-16T01:18:33.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:32 smithi148 ceph-mon[35030]: pgmap v239: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 68 KiB/s wr, 9 op/s 2024-09-16T01:18:33.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:18:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.163+0000 7fc644657640 1 -- 172.21.15.78:0/2497543729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 msgr2=0x7fc63c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.163+0000 7fc644657640 1 --2- 172.21.15.78:0/2497543729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c10ea90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc634009900 tx=0x7fc63402a410 comp rx=0 tx=0).stop 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 -- 172.21.15.78:0/2497543729 shutdown_connections 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 --2- 172.21.15.78:0/2497543729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c10ea90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 --2- 172.21.15.78:0/2497543729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 0x7fc63c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 --2- 172.21.15.78:0/2497543729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 0x7fc63c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 -- 172.21.15.78:0/2497543729 >> 172.21.15.78:0/2497543729 conn(0x7fc63c0fd8c0 msgr2=0x7fc63c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 -- 172.21.15.78:0/2497543729 shutdown_connections 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 -- 172.21.15.78:0/2497543729 wait complete. 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 Processor -- start 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 -- start start 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 0x7fc63c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.164+0000 7fc644657640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 0x7fc63c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c111f90 con 0x7fc63c106f10 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c111e10 con 0x7fc63c101de0 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c112110 con 0x7fc63c102790 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:34.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51650/0 (socket says 172.21.15.78:51650) 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 -- 172.21.15.78:0/1151092272 learned_addr learned my addr 172.21.15.78:0/1151092272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc6423f3640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 0x7fc63c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc641bf2640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 0x7fc63c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 -- 172.21.15.78:0/1151092272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 msgr2=0x7fc63c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 0x7fc63c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 -- 172.21.15.78:0/1151092272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 msgr2=0x7fc63c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 0x7fc63c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 -- 172.21.15.78:0/1151092272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c19d3e0 con 0x7fc63c106f10 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc642bf4640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc634005ab0 tx=0x7fc6340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63403f030 con 0x7fc63c106f10 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc634003ce0 con 0x7fc63c106f10 2024-09-16T01:18:34.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c19d670 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634035070 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.165+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c1a4fb0 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.166+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c1032c0 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.167+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fc634042030 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.168+0000 7fc63b7fe640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 0x7fc6240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.168+0000 7fc6423f3640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 0x7fc6240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.168+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc6340f0820 con 0x7fc63c106f10 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.169+0000 7fc6423f3640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 0x7fc6240a5870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005fd0 tx=0x7fc62c009680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:34.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.172+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc6340bce10 con 0x7fc63c106f10 2024-09-16T01:18:34.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.372+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc63c10a0f0 con 0x7fc63c106f10 2024-09-16T01:18:34.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.372+0000 7fc63b7fe640 1 -- 172.21.15.78:0/1151092272 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fc6340bc710 con 0x7fc63c106f10 2024-09-16T01:18:34.374 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:18:34.374 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:18:34.374 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 msgr2=0x7fc6240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 0x7fc6240a5870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc62c005fd0 tx=0x7fc62c009680 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 msgr2=0x7fc63c19cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc634005ab0 tx=0x7fc6340042f0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 shutdown_connections 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc6240a33b0 0x7fc6240a5870 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc63c106f10 0x7fc63c19cd00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc63c102790 0x7fc63c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 --2- 172.21.15.78:0/1151092272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc63c101de0 0x7fc63c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:34.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 >> 172.21.15.78:0/1151092272 conn(0x7fc63c0fd8c0 msgr2=0x7fc63c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:34.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 shutdown_connections 2024-09-16T01:18:34.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:34.375+0000 7fc644657640 1 -- 172.21.15.78:0/1151092272 wait complete. 2024-09-16T01:18:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:34 smithi078 ceph-mon[30937]: pgmap v240: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-09-16T01:18:35.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:34 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1151092272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:34 smithi193 ceph-mon[35701]: pgmap v240: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-09-16T01:18:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:34 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1151092272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:34 smithi148 ceph-mon[35030]: pgmap v240: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 9 op/s 2024-09-16T01:18:35.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:34 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1151092272' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:18:35.165 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id bc889bfa --release --await' 2024-09-16T01:18:35.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id bc889bfa --release --await 2024-09-16T01:18:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:18:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 440.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 -- 172.21.15.78:0/983176166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102ef0 msgr2=0x7f573810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 --2- 172.21.15.78:0/983176166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102ef0 0x7f573810da40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5730009900 tx=0x7f573002a480 comp rx=0 tx=0).stop 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 -- 172.21.15.78:0/983176166 shutdown_connections 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 --2- 172.21.15.78:0/983176166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102ef0 0x7f573810da40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 --2- 172.21.15.78:0/983176166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738102570 0x7f57381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 --2- 172.21.15.78:0/983176166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738100e20 0x7f5738101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 -- 172.21.15.78:0/983176166 >> 172.21.15.78:0/983176166 conn(0x7f57380fc7d0 msgr2=0x7f57380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 -- 172.21.15.78:0/983176166 shutdown_connections 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.954+0000 7f574044e640 1 -- 172.21.15.78:0/983176166 wait complete. 2024-09-16T01:18:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 Processor -- start 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 -- start start 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738100e20 0x7f5738197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738102ef0 0x7f573819bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5738110f80 con 0x7f5738102570 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5738110e00 con 0x7f5738100e20 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f574044e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5738111100 con 0x7f5738102ef0 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51668/0 (socket says 172.21.15.78:51668) 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 -- 172.21.15.78:0/373011107 learned_addr learned my addr 172.21.15.78:0/373011107 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573e1ea640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738100e20 0x7f5738197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573e9eb640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738102ef0 0x7f573819bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 -- 172.21.15.78:0/373011107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738102ef0 msgr2=0x7f573819bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738102ef0 0x7f573819bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 -- 172.21.15.78:0/373011107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738100e20 msgr2=0x7f5738197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738100e20 0x7f5738197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 -- 172.21.15.78:0/373011107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573819c330 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.955+0000 7f573d9e9640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f572c00d950 tx=0x7f572c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.956+0000 7f57377fe640 1 -- 172.21.15.78:0/373011107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c017070 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.956+0000 7f57377fe640 1 -- 172.21.15.78:0/373011107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572c009c30 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.956+0000 7f57377fe640 1 -- 172.21.15.78:0/373011107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c00f6b0 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.956+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573819c620 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.956+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57381a3f00 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.958+0000 7f57377fe640 1 -- 172.21.15.78:0/373011107 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f572c0109d0 con 0x7f5738102570 2024-09-16T01:18:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.958+0000 7f5742827740 1 Processor -- start 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.958+0000 7f5742827740 1 -- start start 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3be7c0 0x555b4f3caf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3cb450 0x555b4f3cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555b4f323620 con 0x555b4f3bdea0 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555b4f2cae50 con 0x555b4f3cb450 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5742827740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b4f31f5e0 con 0x555b4f3be7c0 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f57377fe640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 0x7f57200a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51672/0 (socket says 172.21.15.78:51672) 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 -- 172.21.15.78:0/3920381875 learned_addr learned my addr 172.21.15.78:0/3920381875 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e1ea640 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3be7c0 0x555b4f3caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 0x7f57200a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573d9e9640 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3cb450 0x555b4f3cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f57377fe640 1 -- 172.21.15.78:0/373011107 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f572c0cfd00 con 0x7f5738102570 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 -- 172.21.15.78:0/3920381875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3be7c0 msgr2=0x555b4f3caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3be7c0 0x555b4f3caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 -- 172.21.15.78:0/3920381875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3cb450 msgr2=0x555b4f3cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3cb450 0x555b4f3cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 -- 172.21.15.78:0/3920381875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4f2bbd40 con 0x555b4f3bdea0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f573e9eb640 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5730004410 tx=0x7f5730004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1785e828d28270a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.959+0000 7f5734dea640 1 -- 172.21.15.78:0/3920381875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573004d070 con 0x555b4f3bdea0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5734dea640 1 -- 172.21.15.78:0/3920381875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730005530 con 0x555b4f3bdea0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5734dea640 1 -- 172.21.15.78:0/3920381875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730035040 con 0x555b4f3bdea0 2024-09-16T01:18:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 -- 172.21.15.78:0/3920381875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 msgr2=0x555b4f3be280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f573e9eb640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 0x7f57200a58f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f573004a000 tx=0x7f573004b000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5730004410 tx=0x7f5730004440 comp rx=0 tx=0).stop 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 -- 172.21.15.78:0/3920381875 shutdown_connections 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3cb450 0x555b4f3cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3be7c0 0x555b4f3caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 --2- 172.21.15.78:0/3920381875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3be280 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 -- 172.21.15.78:0/3920381875 >> 172.21.15.78:0/3920381875 conn(0x555b4f3ba5f0 msgr2=0x555b4f3ba9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 -- 172.21.15.78:0/3920381875 shutdown_connections 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.960+0000 7f5742827740 1 -- 172.21.15.78:0/3920381875 wait complete. 2024-09-16T01:18:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 -- start start 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 Processor -- start 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555b4f3ce020 con 0x555b4f3bdea0 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555b4f2abf10 con 0x555b4f3be7c0 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b4f3ce1a0 con 0x555b4f3cb450 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f5742827740 10 client.0 _ll_register_callbacks cb 0x555b4f2cfdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51680/0 (socket says 172.21.15.78:51680) 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f573e1ea640 1 -- 172.21.15.78:0/3168150275 learned_addr learned my addr 172.21.15.78:0/3168150275 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:35.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.961+0000 7f573d9e9640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 -- 172.21.15.78:0/3168150275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 msgr2=0x555b4f3c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 -- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 msgr2=0x555b4f3c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 -- 172.21.15.78:0/3168150275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4f032580 con 0x555b4f3bdea0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573d9e9640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f573e1ea640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5728011d10 tx=0x7f572800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588436843e1ef132 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728004b20 con 0x555b4f3bdea0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f5728017050 con 0x555b4f3bdea0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5730005530 con 0x555b4f3bdea0 2024-09-16T01:18:35.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5728024050 con 0x555b4f3bdea0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 10 client.16278 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555b4f0fc960 con 0x555b4f3bdea0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730005530 con 0x555b4f3bdea0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5728019030 con 0x555b4f3bdea0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 10 client.16278 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 10 client.16278 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 20 client.16278 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f3db170 0x555b4f3dd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 4 client.16278 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.962+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b4f330b90 con 0x555b4f3db170 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.963+0000 7f573d9e9640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f3db170 0x555b4f3dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.963+0000 7f571dffb640 10 client.16278 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.963+0000 7f573d9e9640 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f3db170 0x555b4f3dd630 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44264a3bcb0f3433 server_cookie=de7ae37331ed4e96 in_seq=0 out_seq=0 2024-09-16T01:18:35.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f571dffb640 1 -- 172.21.15.78:0/3168150275 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555b4f330b90 con 0x555b4f3db170 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f571dffb640 10 client.16278 handle_command_reply: tid=0 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 client.16278 shutdown 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 msgr2=0x555b4f3c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5728011d10 tx=0x7f572800f790 comp rx=0 tx=0).stop 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 shutdown_connections 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f3db170 0x555b4f3dd630 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f571dffb640 0 client.16278 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 --2- 172.21.15.78:0/3168150275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c8400 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 >> 172.21.15.78:0/3168150275 conn(0x555b4f3ba5f0 msgr2=0x555b4f3b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 shutdown_connections 2024-09-16T01:18:35.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 1 -- 172.21.15.78:0/3168150275 wait complete. 2024-09-16T01:18:35.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.964+0000 7f5742827740 20 client.16278 trim_cache size 0 max 16384 2024-09-16T01:18:35.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 Processor -- start 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 -- start start 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3d3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3d3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f57381af740 con 0x555b4f3bdea0 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f57381ad010 con 0x555b4f3be7c0 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f5742827740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f57381aae80 con 0x555b4f3cb450 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51682/0 (socket says 172.21.15.78:51682) 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 -- 172.21.15.78:0/79117941 learned_addr learned my addr 172.21.15.78:0/79117941 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e9eb640 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3d3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 -- 172.21.15.78:0/79117941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 msgr2=0x555b4f3d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3d3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 -- 172.21.15.78:0/79117941 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 msgr2=0x555b4f3d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3d3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 -- 172.21.15.78:0/79117941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4f3c3220 con 0x555b4f3bdea0 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e9eb640 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3d3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f573e1ea640 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f572801a5e0 tx=0x7f57280093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e41cb7cca272001 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.981+0000 7f571ffff640 1 -- 172.21.15.78:0/79117941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572801a660 con 0x555b4f3bdea0 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f571ffff640 1 -- 172.21.15.78:0/79117941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572802d070 con 0x555b4f3bdea0 2024-09-16T01:18:35.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f571ffff640 1 -- 172.21.15.78:0/79117941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728028430 con 0x555b4f3bdea0 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 -- 172.21.15.78:0/79117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 msgr2=0x555b4f3b9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f572801a5e0 tx=0x7f57280093a0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 -- 172.21.15.78:0/79117941 shutdown_connections 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3d3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3d3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 --2- 172.21.15.78:0/79117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3b9a10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 -- 172.21.15.78:0/79117941 >> 172.21.15.78:0/79117941 conn(0x555b4f3ba5f0 msgr2=0x555b4f3d62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 -- 172.21.15.78:0/79117941 shutdown_connections 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.982+0000 7f5742827740 1 -- 172.21.15.78:0/79117941 wait complete. 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 -- start start 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 Processor -- start 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f57381ab4d0 con 0x555b4f3bdea0 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f57381ac940 con 0x555b4f3be7c0 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f57381abba0 con 0x555b4f3cb450 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f5742827740 10 client.0 _ll_register_callbacks cb 0x7f5728011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51684/0 (socket says 172.21.15.78:51684) 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 -- 172.21.15.78:0/1550994551 learned_addr learned my addr 172.21.15.78:0/1550994551 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573d9e9640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e9eb640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 -- 172.21.15.78:0/1550994551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 msgr2=0x555b4f3c18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 -- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 msgr2=0x555b4f3c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e1ea640 1 -- 172.21.15.78:0/1550994551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4f447f80 con 0x555b4f3bdea0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.983+0000 7f573e9eb640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f573e1ea640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f572801a470 tx=0x7f572800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cdb68ee7e466ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728037070 con 0x555b4f3bdea0 2024-09-16T01:18:35.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f572801a660 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f572802d070 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5728043050 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 10 client.16284 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555b4f449230 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572802d070 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5728019040 con 0x555b4f3bdea0 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 10 client.16284 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 10 client.16284 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 20 client.16284 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f449810 0x555b4f44bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 4 client.16284 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bc889bfa", "release": true, "await": true}] 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.984+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bc889bfa", "release": true, "await": true}) -- 0x555b4f32b0a0 con 0x555b4f449810 2024-09-16T01:18:35.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.985+0000 7f573d9e9640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f449810 0x555b4f44bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:35.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.985+0000 7f571dffb640 10 client.16284 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:35.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:35.985+0000 7f573d9e9640 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f449810 0x555b4f44bc30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df7f6589cbcc6da2 server_cookie=5a3eb1dcab24e40d in_seq=0 out_seq=0 2024-09-16T01:18:36.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f571dffb640 1 -- 172.21.15.78:0/1550994551 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x555b4f32b0a0 con 0x555b4f449810 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f571dffb640 10 client.16284 handle_command_reply: tid=0 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 client.16284 shutdown 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 msgr2=0x555b4f3c0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f572801a470 tx=0x7f572800f790 comp rx=0 tx=0).stop 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 shutdown_connections 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555b4f449810 0x555b4f44bc30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b4f3cb450 0x555b4f3c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f571dffb640 0 client.16284 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555b4f3be7c0 0x555b4f3c1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 --2- 172.21.15.78:0/1550994551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b4f3bdea0 0x555b4f3c0e40 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.045+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 >> 172.21.15.78:0/1550994551 conn(0x555b4f3ba5f0 msgr2=0x555b4f3d6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 shutdown_connections 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f5742827740 1 -- 172.21.15.78:0/1550994551 wait complete. 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f5742827740 20 client.16284 trim_cache size 0 max 16384 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:18:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 28, 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "bc889bfa": { 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 28, 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:18:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 143.7, 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:18:36.052 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:18:36.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 msgr2=0x7f57200a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:36.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 0x7f57200a58f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f573004a000 tx=0x7f573004b000 comp rx=0 tx=0).stop 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 msgr2=0x7f57381978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.046+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f572c00d950 tx=0x7f572c00de10 comp rx=0 tx=0).stop 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 shutdown_connections 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f57200a3430 0x7f57200a58f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5738102ef0 0x7f573819bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5738102570 0x7f57381978c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 --2- 172.21.15.78:0/373011107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5738100e20 0x7f5738197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 >> 172.21.15.78:0/373011107 conn(0x7f57380fc7d0 msgr2=0x7f5738105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.049+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 shutdown_connections 2024-09-16T01:18:36.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:18:36.050+0000 7f574044e640 1 -- 172.21.15.78:0/373011107 wait complete. 2024-09-16T01:18:36.771 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bc889bfa', total seconds elapsed: 27.1 2024-09-16T01:18:37.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:36 smithi078 ceph-mon[30937]: pgmap v241: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-09-16T01:18:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:36 smithi193 ceph-mon[35701]: pgmap v241: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-09-16T01:18:37.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:36 smithi148 ceph-mon[35030]: pgmap v241: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-09-16T01:18:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.318+0000 7fc597b8d640 1 -- 172.21.15.78:0/1847316491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901095d0 msgr2=0x7fc590109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.318+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1847316491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901095d0 0x7fc590109a10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc584009900 tx=0x7fc58402a440 comp rx=0 tx=0).stop 2024-09-16T01:18:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.323+0000 7fc597b8d640 1 -- 172.21.15.78:0/1847316491 shutdown_connections 2024-09-16T01:18:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.323+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1847316491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc59011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.323+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1847316491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901095d0 0x7fc590109a10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.323+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1847316491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901091f0 0x7fc590110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.323+0000 7fc597b8d640 1 -- 172.21.15.78:0/1847316491 >> 172.21.15.78:0/1847316491 conn(0x7fc59006ce00 msgr2=0x7fc59006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.329+0000 7fc597b8d640 1 -- 172.21.15.78:0/1847316491 shutdown_connections 2024-09-16T01:18:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.330+0000 7fc597b8d640 1 -- 172.21.15.78:0/1847316491 wait complete. 2024-09-16T01:18:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 Processor -- start 2024-09-16T01:18:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 -- start start 2024-09-16T01:18:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901091f0 0x7fc59011a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901095d0 0x7fc590114930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc590111370 con 0x7fc5901091f0 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901111f0 con 0x7fc59011b140 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.332+0000 7fc597b8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5901114f0 con 0x7fc5901095d0 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.333+0000 7fc59612a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.333+0000 7fc59612a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44396/0 (socket says 172.21.15.78:44396) 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.333+0000 7fc59612a640 1 -- 172.21.15.78:0/1765661016 learned_addr learned my addr 172.21.15.78:0/1765661016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.333+0000 7fc595128640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901095d0 0x7fc590114930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.336+0000 7fc59612a640 1 -- 172.21.15.78:0/1765661016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901095d0 msgr2=0x7fc590114930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.336+0000 7fc59612a640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901095d0 0x7fc590114930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.336+0000 7fc59612a640 1 -- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901091f0 msgr2=0x7fc59011a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.336+0000 7fc59612a640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901091f0 0x7fc59011a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.336+0000 7fc59612a640 1 -- 172.21.15.78:0/1765661016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5901155f0 con 0x7fc59011b140 2024-09-16T01:18:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.337+0000 7fc59612a640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc580009fd0 tx=0x7fc58000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.339+0000 7fc58effd640 1 -- 172.21.15.78:0/1765661016 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580011830 con 0x7fc59011b140 2024-09-16T01:18:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.339+0000 7fc58effd640 1 -- 172.21.15.78:0/1765661016 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc580016070 con 0x7fc59011b140 2024-09-16T01:18:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.339+0000 7fc58effd640 1 -- 172.21.15.78:0/1765661016 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58000f2e0 con 0x7fc59011b140 2024-09-16T01:18:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.339+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5901bc0b0 con 0x7fc59011b140 2024-09-16T01:18:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.339+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5901bc4c0 con 0x7fc59011b140 2024-09-16T01:18:37.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.344+0000 7fc59aa02740 1 Processor -- start 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.345+0000 7fc59aa02740 1 -- start start 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b577fded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b5780b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561b577622f0 con 0x561b57808cb0 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b5770a270 con 0x561b577fe410 2024-09-16T01:18:37.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc59aa02740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b5775e2b0 con 0x561b577fdaf0 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc595929640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc595929640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44402/0 (socket says 172.21.15.78:44402) 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc595929640 1 -- 172.21.15.78:0/2562617368 learned_addr learned my addr 172.21.15.78:0/2562617368 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.346+0000 7fc58effd640 1 -- 172.21.15.78:0/1765661016 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fc580010380 con 0x7fc59011b140 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.347+0000 7fc595929640 1 -- 172.21.15.78:0/2562617368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 msgr2=0x561b577fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:18:37.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.347+0000 7fc595929640 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b577fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.347+0000 7fc595929640 1 -- 172.21.15.78:0/2562617368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 msgr2=0x561b5780b0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:18:37.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.347+0000 7fc595929640 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b5780b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.347+0000 7fc595929640 1 -- 172.21.15.78:0/2562617368 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b57709340 con 0x561b577fe410 2024-09-16T01:18:37.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.348+0000 7fc58effd640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 0x7fc5780a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.348+0000 7fc58effd640 1 -- 172.21.15.78:0/1765661016 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc5800cf5f0 con 0x7fc59011b140 2024-09-16T01:18:37.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.349+0000 7fc595929640 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d80 tx=0x7fc58800b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8f602609a902cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.351+0000 7fc595128640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 0x7fc5780a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.351+0000 7fc577fff640 1 -- 172.21.15.78:0/2562617368 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58800bd40 con 0x561b577fe410 2024-09-16T01:18:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.351+0000 7fc577fff640 1 -- 172.21.15.78:0/2562617368 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc588017070 con 0x561b577fe410 2024-09-16T01:18:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.352+0000 7fc59aa02740 1 -- 172.21.15.78:0/2562617368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 msgr2=0x561b57808770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.352+0000 7fc59aa02740 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d80 tx=0x7fc58800b420 comp rx=0 tx=0).stop 2024-09-16T01:18:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc595128640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 0x7fc5780a5b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc58402a940 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 -- 172.21.15.78:0/2562617368 shutdown_connections 2024-09-16T01:18:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b5780b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57808770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 --2- 172.21.15.78:0/2562617368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b577fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 -- 172.21.15.78:0/2562617368 >> 172.21.15.78:0/2562617368 conn(0x561b577fa060 msgr2=0x561b577fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.353+0000 7fc59aa02740 1 -- 172.21.15.78:0/2562617368 shutdown_connections 2024-09-16T01:18:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.354+0000 7fc59aa02740 1 -- 172.21.15.78:0/2562617368 wait complete. 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 -- start start 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 Processor -- start 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b57805a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57805fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x561b5780b8d0 con 0x561b577fdaf0 2024-09-16T01:18:37.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.355+0000 7fc59aa02740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b576eb290 con 0x561b577fe410 2024-09-16T01:18:37.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.356+0000 7fc59aa02740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b5780ba50 con 0x561b57808cb0 2024-09-16T01:18:37.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.356+0000 7fc59aa02740 10 client.0 _ll_register_callbacks cb 0x561b5771c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:37.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.356+0000 7fc59612a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.356+0000 7fc59612a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:51630/0 (socket says 172.21.15.78:51630) 2024-09-16T01:18:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.356+0000 7fc59612a640 1 -- 172.21.15.78:0/927183640 learned_addr learned my addr 172.21.15.78:0/927183640 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:37.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc595929640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b57805a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc59612a640 1 -- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 msgr2=0x561b57805fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:18:37.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc59612a640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57805fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc59612a640 1 -- 172.21.15.78:0/927183640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 msgr2=0x561b57805a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc59612a640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b57805a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.357+0000 7fc59612a640 1 -- 172.21.15.78:0/927183640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b57475210 con 0x561b57808cb0 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.358+0000 7fc59612a640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc5800979f0 tx=0x7fc58009bc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6187785003bb6369 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.358+0000 7fc5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.358+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800a4200 con 0x561b57808cb0 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.359+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc580097c00 con 0x561b57808cb0 2024-09-16T01:18:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.359+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc588017070 con 0x561b57808cb0 2024-09-16T01:18:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.360+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc5800ccb70 con 0x561b57808cb0 2024-09-16T01:18:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.360+0000 7fc59aa02740 10 client.25723 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.360+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b57541620 con 0x561b57808cb0 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc588017070 con 0x561b57808cb0 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc58009a9e0 con 0x561b57808cb0 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 10 client.25723 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 10 client.25723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 20 client.25723 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57818bc0 0x561b5781aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 4 client.25723 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b5776fa20 con 0x561b57818bc0 2024-09-16T01:18:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.361+0000 7fc595929640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57818bc0 0x561b5781aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.362+0000 7fc5757fa640 10 client.25723 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.362+0000 7fc595929640 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57818bc0 0x561b5781aff0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5beaac31a81ecfb server_cookie=69401eeb3c5ce522 in_seq=0 out_seq=0 2024-09-16T01:18:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.363+0000 7fc5757fa640 1 -- 172.21.15.78:0/927183640 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b5776fa20 con 0x561b57818bc0 2024-09-16T01:18:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.363+0000 7fc5757fa640 10 client.25723 handle_command_reply: tid=0 2024-09-16T01:18:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.363+0000 7fc59aa02740 1 client.25723 shutdown 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 msgr2=0x561b57806510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc5800979f0 tx=0x7fc58009bc90 comp rx=0 tx=0).stop 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 shutdown_connections 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57818bc0 0x561b5781aff0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b57808cb0 0x561b57806510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57805fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 --2- 172.21.15.78:0/927183640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b57805a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 >> 172.21.15.78:0/927183640 conn(0x561b577fa060 msgr2=0x561b577ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc5757fa640 0 client.25723 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 shutdown_connections 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 1 -- 172.21.15.78:0/927183640 wait complete. 2024-09-16T01:18:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.365+0000 7fc59aa02740 20 client.25723 trim_cache size 0 max 16384 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 Processor -- start 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 -- start start 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b57800560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57804340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0129e0 con 0x561b57808cb0 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51736/0 (socket says 172.21.15.78:51736) 2024-09-16T01:18:37.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 -- 172.21.15.78:0/3146172373 learned_addr learned my addr 172.21.15.78:0/3146172373 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c0102b0 con 0x561b577fe410 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c00e120 con 0x561b577fdaf0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc595929640 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b57800560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc595128640 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57804340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 -- 172.21.15.78:0/3146172373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 msgr2=0x561b57800560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b57800560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 -- 172.21.15.78:0/3146172373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 msgr2=0x561b57804340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57804340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.383+0000 7fc59612a640 1 -- 172.21.15.78:0/3146172373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b577f9e90 con 0x561b57808cb0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.384+0000 7fc59612a640 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc58009be40 tx=0x7fc58009fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b961cd18a24497a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.384+0000 7fc5777fe640 1 -- 172.21.15.78:0/3146172373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800a9070 con 0x561b57808cb0 2024-09-16T01:18:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.384+0000 7fc5777fe640 1 -- 172.21.15.78:0/3146172373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5800ccd10 con 0x561b57808cb0 2024-09-16T01:18:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.384+0000 7fc5777fe640 1 -- 172.21.15.78:0/3146172373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58009a9e0 con 0x561b57808cb0 2024-09-16T01:18:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 msgr2=0x561b57806340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc58009be40 tx=0x7fc58009fa00 comp rx=0 tx=0).stop 2024-09-16T01:18:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 shutdown_connections 2024-09-16T01:18:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b57808cb0 0x561b57806340 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b577fe410 0x561b57804340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 --2- 172.21.15.78:0/3146172373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fdaf0 0x561b57800560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 >> 172.21.15.78:0/3146172373 conn(0x561b577fa060 msgr2=0x561b57807b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 shutdown_connections 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- 172.21.15.78:0/3146172373 wait complete. 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 -- start start 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.385+0000 7fc59aa02740 1 Processor -- start 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fe410 0x561b5780c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 0x561b5780ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c00e770 con 0x561b577fdaf0 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc595929640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc595929640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51744/0 (socket says 172.21.15.78:51744) 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc595929640 1 -- 172.21.15.78:0/151540660 learned_addr learned my addr 172.21.15.78:0/151540660 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c00fbe0 con 0x561b57808cb0 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc595128640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fe410 0x561b5780c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc57c00ee40 con 0x561b577fe410 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59612a640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 0x561b5780ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.386+0000 7fc59aa02740 10 client.0 _ll_register_callbacks cb 0x7fc580097190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 -- 172.21.15.78:0/151540660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fe410 msgr2=0x561b5780c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fe410 0x561b5780c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 -- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 msgr2=0x561b5780ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 0x561b5780ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b577f52d0 con 0x561b577fdaf0 2024-09-16T01:18:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc59612a640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 0x561b5780ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc595929640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc588000f80 tx=0x7fc588009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829ba0e2e452ba7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.387+0000 7fc5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58802b070 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7fc588009870 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b57769d70 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc588012430 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc59aa02740 10 client.16296 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b578833e0 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.388+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561b57769d70 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc588011070 con 0x561b577fdaf0 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 10 client.16296 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 20 client.16296 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57883880 0x561b57885c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 4 client.16296 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b576e6900 con 0x561b57883880 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.389+0000 7fc595128640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57883880 0x561b57885c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.390+0000 7fc595128640 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57883880 0x561b57885c70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8569a5a0204d14a4 server_cookie=876f684ca895a214 in_seq=0 out_seq=0 2024-09-16T01:18:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.390+0000 7fc5757fa640 10 client.16296 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.397+0000 7fc5757fa640 1 -- 172.21.15.78:0/151540660 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b576e6900 con 0x561b57883880 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.397+0000 7fc5757fa640 10 client.16296 handle_command_reply: tid=0 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.397+0000 7fc59aa02740 1 client.16296 shutdown 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 msgr2=0x561b578111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc588000f80 tx=0x7fc588009780 comp rx=0 tx=0).stop 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 shutdown_connections 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x561b57883880 0x561b57885c70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b57808cb0 0x561b5780ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b577fe410 0x561b5780c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 --2- 172.21.15.78:0/151540660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561b577fdaf0 0x561b578111a0 secure :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc588000f80 tx=0x7fc588009780 comp rx=0 tx=0).stop 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 >> 172.21.15.78:0/151540660 conn(0x561b577fa060 msgr2=0x561b577f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc5757fa640 0 client.16296 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 shutdown_connections 2024-09-16T01:18:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 1 -- 172.21.15.78:0/151540660 wait complete. 2024-09-16T01:18:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.398+0000 7fc59aa02740 20 client.16296 trim_cache size 0 max 16384 2024-09-16T01:18:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:18:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:18:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:18:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:18:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 msgr2=0x7fc5780a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 0x7fc5780a5b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc584000c00 tx=0x7fc58402a940 comp rx=0 tx=0).stop 2024-09-16T01:18:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 msgr2=0x7fc590114e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc580009fd0 tx=0x7fc58000b8e0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 shutdown_connections 2024-09-16T01:18:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc5780a3690 0x7fc5780a5b50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc59011b140 0x7fc590114e70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5901095d0 0x7fc590114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 --2- 172.21.15.78:0/1765661016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5901091f0 0x7fc59011a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 >> 172.21.15.78:0/1765661016 conn(0x7fc59006ce00 msgr2=0x7fc590071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 shutdown_connections 2024-09-16T01:18:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:37.399+0000 7fc597b8d640 1 -- 172.21.15.78:0/1765661016 wait complete. 2024-09-16T01:18:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:38 smithi193 ceph-mon[35701]: pgmap v242: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 KiB/s wr, 13 op/s 2024-09-16T01:18:39.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:38 smithi148 ceph-mon[35030]: pgmap v242: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 KiB/s wr, 13 op/s 2024-09-16T01:18:39.240 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:38 smithi078 ceph-mon[30937]: pgmap v242: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 KiB/s wr, 13 op/s 2024-09-16T01:18:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 445.0 s, 601.1 tps, lat 903.964 ms stddev 4076.595 2024-09-16T01:18:41.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:40 smithi078 ceph-mon[30937]: pgmap v243: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 549 KiB/s wr, 77 op/s 2024-09-16T01:18:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:40 smithi193 ceph-mon[35701]: pgmap v243: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 549 KiB/s wr, 77 op/s 2024-09-16T01:18:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:40 smithi148 ceph-mon[35030]: pgmap v243: 129 pgs: 129 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 549 KiB/s wr, 77 op/s 2024-09-16T01:18:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:42 smithi193 ceph-mon[35701]: pgmap v244: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-16T01:18:43.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:42 smithi148 ceph-mon[35030]: pgmap v244: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-16T01:18:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:42 smithi078 ceph-mon[30937]: pgmap v244: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-09-16T01:18:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:44 smithi148 ceph-mon[35030]: pgmap v245: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-09-16T01:18:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:44 smithi078 ceph-mon[30937]: pgmap v245: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-09-16T01:18:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:44 smithi193 ceph-mon[35701]: pgmap v245: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.0 MiB/s wr, 214 op/s 2024-09-16T01:18:45.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 450.0 s, 930.6 tps, lat 113.493 ms stddev 171.204 2024-09-16T01:18:47.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:46 smithi148 ceph-mon[35030]: pgmap v246: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 4.2 MiB/s wr, 401 op/s 2024-09-16T01:18:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:46 smithi078 ceph-mon[30937]: pgmap v246: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 4.2 MiB/s wr, 401 op/s 2024-09-16T01:18:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:46 smithi193 ceph-mon[35701]: pgmap v246: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 4.2 MiB/s wr, 401 op/s 2024-09-16T01:18:49.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:48 smithi148 ceph-mon[35030]: pgmap v247: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.6 MiB/s wr, 444 op/s 2024-09-16T01:18:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:48 smithi078 ceph-mon[30937]: pgmap v247: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.6 MiB/s wr, 444 op/s 2024-09-16T01:18:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:48 smithi193 ceph-mon[35701]: pgmap v247: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 4.6 MiB/s wr, 444 op/s 2024-09-16T01:18:50.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.538+0000 7f6475d9b640 1 -- 172.21.15.78:0/1272679819 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64680318d0 con 0x7f64781069e0 2024-09-16T01:18:50.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.539+0000 7f647cd66640 1 -- 172.21.15.78:0/1272679819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 msgr2=0x7f6478106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.539+0000 7f647cd66640 1 --2- 172.21.15.78:0/1272679819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f6478106dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6468009900 tx=0x7f646802a440 comp rx=0 tx=0).stop 2024-09-16T01:18:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.540+0000 7f647cd66640 1 -- 172.21.15.78:0/1272679819 shutdown_connections 2024-09-16T01:18:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.540+0000 7f647cd66640 1 --2- 172.21.15.78:0/1272679819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f647810d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.540+0000 7f647cd66640 1 --2- 172.21.15.78:0/1272679819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 0x7f6478100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.540+0000 7f647cd66640 1 --2- 172.21.15.78:0/1272679819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f6478106dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.540+0000 7f647cd66640 1 -- 172.21.15.78:0/1272679819 >> 172.21.15.78:0/1272679819 conn(0x7f64780fc6b0 msgr2=0x7f64780fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.542+0000 7f647cd66640 1 -- 172.21.15.78:0/1272679819 shutdown_connections 2024-09-16T01:18:50.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.542+0000 7f647cd66640 1 -- 172.21.15.78:0/1272679819 wait complete. 2024-09-16T01:18:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.543+0000 7f647cd66640 1 Processor -- start 2024-09-16T01:18:50.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.543+0000 7f647cd66640 1 -- start start 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 0x7f6478116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f64781166d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f64781112a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f647810fd40 con 0x7f6478101370 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f647810fbc0 con 0x7f64781069e0 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647cd66640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f647810fec0 con 0x7f64781009d0 2024-09-16T01:18:50.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f647659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 0x7f6478116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f64781112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f64781112a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:49222/0 (socket says 172.21.15.78:49222) 2024-09-16T01:18:50.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f6476d9d640 1 -- 172.21.15.78:0/4002356294 learned_addr learned my addr 172.21.15.78:0/4002356294 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:50.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.544+0000 7f646ffff640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f64781166d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 -- 172.21.15.78:0/4002356294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 msgr2=0x7f6478116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 0x7f6478116190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 -- 172.21.15.78:0/4002356294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 msgr2=0x7f64781112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f64781112a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 -- 172.21.15.78:0/4002356294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6478111b60 con 0x7f6478101370 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.545+0000 7f646ffff640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f64781166d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f646000d6d0 tx=0x7f646000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.546+0000 7f646f7fe640 1 -- 172.21.15.78:0/4002356294 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460017070 con 0x7f6478101370 2024-09-16T01:18:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.546+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6478111e50 con 0x7f6478101370 2024-09-16T01:18:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.546+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647811c080 con 0x7f6478101370 2024-09-16T01:18:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.546+0000 7f646f7fe640 1 -- 172.21.15.78:0/4002356294 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6460009820 con 0x7f6478101370 2024-09-16T01:18:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.547+0000 7f646f7fe640 1 -- 172.21.15.78:0/4002356294 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460004ce0 con 0x7f6478101370 2024-09-16T01:18:50.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.550+0000 7f646f7fe640 1 -- 172.21.15.78:0/4002356294 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6460005220 con 0x7f6478101370 2024-09-16T01:18:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.551+0000 7f646f7fe640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 0x7f64580a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.551+0000 7f646f7fe640 1 -- 172.21.15.78:0/4002356294 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f64600d2170 con 0x7f6478101370 2024-09-16T01:18:50.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.551+0000 7f647659c640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 0x7f64580a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.554+0000 7f647659c640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 0x7f64580a5920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6468005ab0 tx=0x7f6468005fd0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.558+0000 7f647fbdf740 1 Processor -- start 2024-09-16T01:18:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 -- start start 2024-09-16T01:18:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b5eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b69770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55da12ac32f0 con 0x55da12b5eaf0 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55da12a6b270 con 0x55da12b69cb0 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f647fbdf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da12abf2b0 con 0x55da12b5f410 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:49228/0 (socket says 172.21.15.78:49228) 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f6476d9d640 1 -- 172.21.15.78:0/3709585000 learned_addr learned my addr 172.21.15.78:0/3709585000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.559+0000 7f646ffff640 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b69770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 -- 172.21.15.78:0/3709585000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 msgr2=0x55da12b69770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b69770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 -- 172.21.15.78:0/3709585000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 msgr2=0x55da12b5eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b5eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 -- 172.21.15.78:0/3709585000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da12a6a340 con 0x55da12b69cb0 2024-09-16T01:18:50.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f6476d9d640 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fa0 tx=0x7f6470007800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d369991625dfebb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f646cff9640 1 -- 172.21.15.78:0/3709585000 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470019070 con 0x55da12b69cb0 2024-09-16T01:18:50.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f646cff9640 1 -- 172.21.15.78:0/3709585000 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470012bf0 con 0x55da12b69cb0 2024-09-16T01:18:50.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.560+0000 7f646cff9640 1 -- 172.21.15.78:0/3709585000 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470014e10 con 0x55da12b69cb0 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.561+0000 7f647fbdf740 1 -- 172.21.15.78:0/3709585000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 msgr2=0x55da12b6c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.561+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fa0 tx=0x7f6470007800 comp rx=0 tx=0).stop 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 -- 172.21.15.78:0/3709585000 shutdown_connections 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b6c0a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b69770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3709585000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b5eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 -- 172.21.15.78:0/3709585000 >> 172.21.15.78:0/3709585000 conn(0x55da12b5b060 msgr2=0x55da12b5b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 -- 172.21.15.78:0/3709585000 shutdown_connections 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.562+0000 7f647fbdf740 1 -- 172.21.15.78:0/3709585000 wait complete. 2024-09-16T01:18:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647fbdf740 1 -- start start 2024-09-16T01:18:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647fbdf740 1 Processor -- start 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 0x55da12b67050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:47720/0 (socket says 172.21.15.78:47720) 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f646ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 0x55da12b67050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f646ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 0x55da12b67050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44678/0 (socket says 172.21.15.78:44678) 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b67590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.563+0000 7f647659c640 1 -- 172.21.15.78:0/3963449687 learned_addr learned my addr 172.21.15.78:0/3963449687 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55da12b6c880 con 0x55da12b5f410 2024-09-16T01:18:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55da12a4c290 con 0x55da12b69cb0 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da12b6ca00 con 0x55da12b5eaf0 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f6476d9d640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b67590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647fbdf740 10 client.0 _ll_register_callbacks cb 0x55da12a7d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 -- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 msgr2=0x55da12b67590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b67590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 -- 172.21.15.78:0/3963449687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 msgr2=0x55da12b67050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 0x55da12b67050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da127d6210 con 0x55da12b5eaf0 2024-09-16T01:18:50.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f647659c640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6468009900 tx=0x7f6468004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2387ff18dd0e805 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.564+0000 7f6465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:18:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646804d020 con 0x55da12b5eaf0 2024-09-16T01:18:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6470012bf0 con 0x55da12b5eaf0 2024-09-16T01:18:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f6468004b90 con 0x55da12b5eaf0 2024-09-16T01:18:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6468058050 con 0x55da12b5eaf0 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 10 client.25741 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da128a2620 con 0x55da12b5eaf0 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470012bf0 con 0x55da12b5eaf0 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6468032500 con 0x55da12b5eaf0 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 10 client.25741 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 10 client.25741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 20 client.25741 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12b7c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 4 client.25741 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.565+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da12ad0a20 con 0x55da12b79c30 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.566+0000 7f646ffff640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12b7c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.566+0000 7f6465ffb640 10 client.25741 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.566+0000 7f646ffff640 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12b7c0c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8ec1298a1c4240b server_cookie=1f0b1236238588c0 in_seq=0 out_seq=0 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.567+0000 7f6465ffb640 1 -- 172.21.15.78:0/3963449687 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da12ad0a20 con 0x55da12b79c30 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.567+0000 7f6465ffb640 10 client.25741 handle_command_reply: tid=0 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.567+0000 7f647fbdf740 1 client.25741 shutdown 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.567+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 msgr2=0x55da12b66b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.567+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6468009900 tx=0x7f6468004580 comp rx=0 tx=0).stop 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 shutdown_connections 2024-09-16T01:18:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12b7c0c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f6465ffb640 0 client.25741 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b67590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5f410 0x55da12b67050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3963449687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5eaf0 0x55da12b66b10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 >> 172.21.15.78:0/3963449687 conn(0x55da12b5b060 msgr2=0x55da12b60dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 shutdown_connections 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 1 -- 172.21.15.78:0/3963449687 wait complete. 2024-09-16T01:18:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.568+0000 7f647fbdf740 20 client.25741 trim_cache size 0 max 16384 2024-09-16T01:18:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.585+0000 7f647fbdf740 1 Processor -- start 2024-09-16T01:18:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.585+0000 7f647fbdf740 1 -- start start 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 0x55da12b673c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 0x55da12b81f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f645c012880 con 0x55da12b69cb0 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f645c010150 con 0x55da12b5eaf0 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647fbdf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f645c00dfc0 con 0x55da12b5f410 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f646ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 0x55da12b673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 0x55da12b81f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f646ffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:47736/0 (socket says 172.21.15.78:47736) 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f647659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 0x55da12b673c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:49246/0 (socket says 172.21.15.78:49246) 2024-09-16T01:18:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f646ffff640 1 -- 172.21.15.78:0/1464959080 learned_addr learned my addr 172.21.15.78:0/1464959080 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 -- 172.21.15.78:0/1464959080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 msgr2=0x55da12b819d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 -- 172.21.15.78:0/1464959080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 msgr2=0x55da12b673c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 0x55da12b673c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 -- 172.21.15.78:0/1464959080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da12b5ae90 con 0x55da12b69cb0 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f646ffff640 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.586+0000 7f6476d9d640 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 0x55da12b81f10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f6470009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e00aecf05a8e273 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f6467fff640 1 -- 172.21.15.78:0/1464959080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470025020 con 0x55da12b69cb0 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f6467fff640 1 -- 172.21.15.78:0/1464959080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647001a040 con 0x55da12b69cb0 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f6467fff640 1 -- 172.21.15.78:0/1464959080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470019040 con 0x55da12b69cb0 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 -- 172.21.15.78:0/1464959080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 msgr2=0x55da12b81f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 0x55da12b81f10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f6470009c40 comp rx=0 tx=0).stop 2024-09-16T01:18:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 -- 172.21.15.78:0/1464959080 shutdown_connections 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b69cb0 0x55da12b81f10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 --2- 172.21.15.78:0/1464959080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b5eaf0 0x55da12b673c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 -- 172.21.15.78:0/1464959080 >> 172.21.15.78:0/1464959080 conn(0x55da12b5b060 msgr2=0x55da12b68b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 -- 172.21.15.78:0/1464959080 shutdown_connections 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.587+0000 7f647fbdf740 1 -- 172.21.15.78:0/1464959080 wait complete. 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 -- start start 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 Processor -- start 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b63c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f645c00e610 con 0x55da12b5eaf0 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f645c00fa80 con 0x55da12b69cb0 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f645c00ece0 con 0x55da12b5f410 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647fbdf740 10 client.0 _ll_register_callbacks cb 0x7f6468040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44698/0 (socket says 172.21.15.78:44698) 2024-09-16T01:18:50.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f6476d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b63c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 -- 172.21.15.78:0/3093892077 learned_addr learned my addr 172.21.15.78:0/3093892077 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f646ffff640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 -- 172.21.15.78:0/3093892077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 msgr2=0x55da12b636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b636e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 -- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 msgr2=0x55da12b63c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b63c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f647659c640 1 -- 172.21.15.78:0/3093892077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da12b562d0 con 0x55da12b5eaf0 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.588+0000 7f6476d9d640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b63c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647659c640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6468032920 tx=0x7f6468032710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b14095f94acbdb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:18:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646804d070 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f6468032d90 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f647001a040 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6468035040 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 10 client.16308 fetch_fsmap learned FSMap version 31 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da12be4740 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647001a040 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f64680495a0 con 0x55da12b5eaf0 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 10 client.16308 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 20 client.16308 populate_metadata read hostname 'smithi078' 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12be6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 4 client.16308 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:18:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da12acad70 con 0x55da12b79c30 2024-09-16T01:18:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.589+0000 7f646ffff640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12be6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:18:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.590+0000 7f646ffff640 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12be6f30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ef7f1fc748d6cc server_cookie=fd653e000e0203b4 in_seq=0 out_seq=0 2024-09-16T01:18:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.590+0000 7f6465ffb640 10 client.16308 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.596+0000 7f6465ffb640 1 -- 172.21.15.78:0/3093892077 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da12acad70 con 0x55da12b79c30 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.596+0000 7f6465ffb640 10 client.16308 handle_command_reply: tid=0 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.596+0000 7f647fbdf740 1 client.16308 shutdown 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.596+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 msgr2=0x55da12b72380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.596+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6468032920 tx=0x7f6468032710 comp rx=0 tx=0).stop 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 shutdown_connections 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55da12b79c30 0x55da12be6f30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da12b69cb0 0x55da12b63c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da12b5f410 0x55da12b636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 --2- 172.21.15.78:0/3093892077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55da12b5eaf0 0x55da12b72380 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 >> 172.21.15.78:0/3093892077 conn(0x55da12b5b060 msgr2=0x55da12b60550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f6465ffb640 0 client.16308 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 shutdown_connections 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 1 -- 172.21.15.78:0/3093892077 wait complete. 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647fbdf740 20 client.16308 trim_cache size 0 max 16384 2024-09-16T01:18:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:18:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:18:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:18:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:18:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 msgr2=0x7f64580a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 0x7f64580a5920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6468005ab0 tx=0x7f6468005fd0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 msgr2=0x7f64781166d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.597+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f64781166d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f646000d6d0 tx=0x7f646000db90 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 shutdown_connections 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64580a3460 0x7f64580a5920 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64781069e0 0x7f64781112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6478101370 0x7f64781166d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 --2- 172.21.15.78:0/4002356294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64781009d0 0x7f6478116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 >> 172.21.15.78:0/4002356294 conn(0x7f64780fc6b0 msgr2=0x7f64780fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:18:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 shutdown_connections 2024-09-16T01:18:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:18:50.598+0000 7f647cd66640 1 -- 172.21.15.78:0/4002356294 wait complete. 2024-09-16T01:18:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 455.0 s, 1105.8 tps, lat 86.721 ms stddev 101.696 2024-09-16T01:18:51.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:50 smithi148 ceph-mon[35030]: pgmap v248: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.0 MiB/s wr, 558 op/s 2024-09-16T01:18:51.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:50 smithi078 ceph-mon[30937]: pgmap v248: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.0 MiB/s wr, 558 op/s 2024-09-16T01:18:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:50 smithi193 ceph-mon[35701]: pgmap v248: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 6.0 MiB/s wr, 558 op/s 2024-09-16T01:18:53.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:52 smithi148 ceph-mon[35030]: pgmap v249: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.0 MiB/s wr, 627 op/s 2024-09-16T01:18:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:52 smithi078 ceph-mon[30937]: pgmap v249: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.0 MiB/s wr, 627 op/s 2024-09-16T01:18:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:52 smithi193 ceph-mon[35701]: pgmap v249: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.0 MiB/s wr, 627 op/s 2024-09-16T01:18:55.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:54 smithi193 ceph-mon[35701]: pgmap v250: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 504 op/s 2024-09-16T01:18:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:54 smithi148 ceph-mon[35030]: pgmap v250: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 504 op/s 2024-09-16T01:18:55.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:54 smithi078 ceph-mon[30937]: pgmap v250: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.6 MiB/s wr, 504 op/s 2024-09-16T01:18:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 460.0 s, 904.4 tps, lat 115.083 ms stddev 142.244 2024-09-16T01:18:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:56 smithi148 ceph-mon[35030]: pgmap v251: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 645 op/s 2024-09-16T01:18:57.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:56 smithi078 ceph-mon[30937]: pgmap v251: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 645 op/s 2024-09-16T01:18:57.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:56 smithi193 ceph-mon[35701]: pgmap v251: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 645 op/s 2024-09-16T01:18:59.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:58 smithi148 ceph-mon[35030]: pgmap v252: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:18:59.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:58 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:18:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:58 smithi078 ceph-mon[30937]: pgmap v252: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:18:59.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:58 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:18:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:58 smithi193 ceph-mon[35701]: pgmap v252: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:18:59.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:58 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:19:00.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:59 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:19:00.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:59 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:19:00.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:18:59 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:19:00.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:59 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:19:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:59 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:19:00.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:18:59 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:19:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:59 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:19:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:59 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:19:00.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:18:59 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:19:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 465.0 s, 967.6 tps, lat 92.075 ms stddev 99.269 2024-09-16T01:19:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:01 smithi078 ceph-mon[30937]: pgmap v253: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-09-16T01:19:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:01 smithi193 ceph-mon[35701]: pgmap v253: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-09-16T01:19:01.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:01 smithi148 ceph-mon[35030]: pgmap v253: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-09-16T01:19:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:03 smithi078 ceph-mon[30937]: pgmap v254: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 648 op/s 2024-09-16T01:19:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:03 smithi193 ceph-mon[35701]: pgmap v254: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 648 op/s 2024-09-16T01:19:03.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:03 smithi148 ceph-mon[35030]: pgmap v254: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 648 op/s 2024-09-16T01:19:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.739+0000 7f914a85e640 1 -- 172.21.15.78:0/2080878457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9144108d10 msgr2=0x7f914410cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.739+0000 7f914a85e640 1 --2- 172.21.15.78:0/2080878457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9144108d10 0x7f914410cd50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f913c009ca0 tx=0x7f913c02a8f0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 -- 172.21.15.78:0/2080878457 shutdown_connections 2024-09-16T01:19:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 --2- 172.21.15.78:0/2080878457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f914410d4b0 0x7f9144115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 --2- 172.21.15.78:0/2080878457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9144108d10 0x7f914410cd50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 --2- 172.21.15.78:0/2080878457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9144108360 0x7f9144108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 -- 172.21.15.78:0/2080878457 >> 172.21.15.78:0/2080878457 conn(0x7f9144077ad0 msgr2=0x7f9144079ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 -- 172.21.15.78:0/2080878457 shutdown_connections 2024-09-16T01:19:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.740+0000 7f914a85e640 1 -- 172.21.15.78:0/2080878457 wait complete. 2024-09-16T01:19:03.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.741+0000 7f914a85e640 1 Processor -- start 2024-09-16T01:19:03.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.741+0000 7f914a85e640 1 -- start start 2024-09-16T01:19:03.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.741+0000 7f914a85e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f914a85e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914410d4b0 0x7f9144082c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f9143fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f9143fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34844/0 (socket says 172.21.15.78:34844) 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f9143fff640 1 -- 172.21.15.78:0/2945733571 learned_addr learned my addr 172.21.15.78:0/2945733571 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f914407b740 0x7f914407bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f91441185a0 con 0x7f914410d4b0 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9144118420 con 0x7f914407b740 2024-09-16T01:19:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.742+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9144118720 con 0x7f9144108360 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 -- 172.21.15.78:0/2945733571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f914407b740 msgr2=0x7f914407bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f914407b740 0x7f914407bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 -- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914410d4b0 msgr2=0x7f9144082c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914410d4b0 0x7f9144082c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914407c4b0 con 0x7f9144108360 2024-09-16T01:19:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.743+0000 7f9143fff640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f913800c920 tx=0x7f913800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.744+0000 7f91417fa640 1 -- 172.21.15.78:0/2945733571 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138007c70 con 0x7f9144108360 2024-09-16T01:19:03.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.744+0000 7f91417fa640 1 -- 172.21.15.78:0/2945733571 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9138007e10 con 0x7f9144108360 2024-09-16T01:19:03.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.744+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9144080240 con 0x7f9144108360 2024-09-16T01:19:03.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.744+0000 7f91417fa640 1 -- 172.21.15.78:0/2945733571 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138018890 con 0x7f9144108360 2024-09-16T01:19:03.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.744+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9144080790 con 0x7f9144108360 2024-09-16T01:19:03.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.748+0000 7f91417fa640 1 -- 172.21.15.78:0/2945733571 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f9138018ab0 con 0x7f9144108360 2024-09-16T01:19:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.755+0000 7f914d5fc740 1 Processor -- start 2024-09-16T01:19:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 -- start start 2024-09-16T01:19:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df11ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 0x5640df127770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df127cb0 0x5640df12a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5640df0812f0 con 0x5640df11caf0 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5640df029270 con 0x5640df11d410 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.758+0000 7f914d5fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640df07d2b0 con 0x5640df127cb0 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df11ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9143fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 0x5640df127770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9143fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 0x5640df127770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33612/0 (socket says 172.21.15.78:33612) 2024-09-16T01:19:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9143fff640 1 -- 172.21.15.78:0/1327405900 learned_addr learned my addr 172.21.15.78:0/1327405900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 -- 172.21.15.78:0/1327405900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df127cb0 msgr2=0x5640df12a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df127cb0 0x5640df12a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 -- 172.21.15.78:0/1327405900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 msgr2=0x5640df127770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 0x5640df127770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.759+0000 7f9148dfb640 1 -- 172.21.15.78:0/1327405900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640df028340 con 0x5640df11caf0 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.760+0000 7f9148dfb640 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df11ced0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9134009670 tx=0x7f9134009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f996970734d450d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.760+0000 7f912a7fc640 1 -- 172.21.15.78:0/1327405900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134015020 con 0x5640df11caf0 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.760+0000 7f912a7fc640 1 -- 172.21.15.78:0/1327405900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f913400bda0 con 0x5640df11caf0 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.760+0000 7f912a7fc640 1 -- 172.21.15.78:0/1327405900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134010380 con 0x5640df11caf0 2024-09-16T01:19:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.761+0000 7f914d5fc740 1 -- 172.21.15.78:0/1327405900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 msgr2=0x5640df11ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.761+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df11ced0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9134009670 tx=0x7f9134009b30 comp rx=0 tx=0).stop 2024-09-16T01:19:03.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.762+0000 7f91417fa640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 0x7f912c0a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f91417fa640 1 -- 172.21.15.78:0/2945733571 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f91380d02d0 con 0x7f9144108360 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f914d5fc740 1 -- 172.21.15.78:0/1327405900 shutdown_connections 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df127cb0 0x5640df12a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11d410 0x5640df127770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1327405900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df11ced0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f914d5fc740 1 -- 172.21.15.78:0/1327405900 >> 172.21.15.78:0/1327405900 conn(0x5640df119060 msgr2=0x5640df119440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.763+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 0x7f912c0a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.764+0000 7f914d5fc740 1 -- 172.21.15.78:0/1327405900 shutdown_connections 2024-09-16T01:19:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.764+0000 7f914d5fc740 1 -- 172.21.15.78:0/1327405900 wait complete. 2024-09-16T01:19:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.765+0000 7f914d5fc740 1 -- start start 2024-09-16T01:19:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.765+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 0x7f912c0a5a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9134011a80 tx=0x7f9134000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.765+0000 7f914d5fc740 1 Processor -- start 2024-09-16T01:19:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df124a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df124fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df125510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f91437fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df124fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f91437fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df124fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34862/0 (socket says 172.21.15.78:34862) 2024-09-16T01:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f91437fe640 1 -- 172.21.15.78:0/2142077870 learned_addr learned my addr 172.21.15.78:0/2142077870 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.766+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df125510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.767+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5640df12a8d0 con 0x5640df11caf0 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.767+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5640df00a290 con 0x5640df127cb0 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.767+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5640df12aa50 con 0x5640df11d410 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.767+0000 7f914d5fc740 10 client.0 _ll_register_callbacks cb 0x5640df03b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.767+0000 7f9143fff640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df124a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.768+0000 7f9148dfb640 1 -- 172.21.15.78:0/2142077870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 msgr2=0x5640df124fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.768+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df124fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.768+0000 7f9148dfb640 1 -- 172.21.15.78:0/2142077870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 msgr2=0x5640df124a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.768+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df124a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.768+0000 7f9148dfb640 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640ded94210 con 0x5640df127cb0 2024-09-16T01:19:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.770+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df125510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f913400a010 tx=0x7f9134011ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=273b62dbbac1917d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.771 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.770+0000 7f91237fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.772+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913402a070 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.773+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f913401ad80 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.773+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f913400bda0 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.773+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9134015040 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.773+0000 7f914d5fc740 10 client.25724 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.773+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640dee60620 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f913400bda0 con 0x5640df127cb0 2024-09-16T01:19:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f91340394b0 con 0x5640df127cb0 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 10 client.25724 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 10 client.25724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 20 client.25724 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 4 client.25724 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.774+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640df08ea20 con 0x5640df137b80 2024-09-16T01:19:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.775+0000 7f9143fff640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.779+0000 7f91237fe640 10 client.25724 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.779+0000 7f9143fff640 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df139f70 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cddb5224bc71cfb2 server_cookie=663f26855ef8e291 in_seq=0 out_seq=0 2024-09-16T01:19:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.779+0000 7f91237fe640 1 -- 172.21.15.78:0/2142077870 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640df08ea20 con 0x5640df137b80 2024-09-16T01:19:03.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f91237fe640 10 client.25724 handle_command_reply: tid=0 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 client.25724 shutdown 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 msgr2=0x5640df125510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df125510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f913400a010 tx=0x7f9134011ae0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 shutdown_connections 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df139f70 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df125510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f91237fe640 0 client.25724 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df124fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2142077870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df124a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 >> 172.21.15.78:0/2142077870 conn(0x5640df119060 msgr2=0x5640df11edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 shutdown_connections 2024-09-16T01:19:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.780+0000 7f914d5fc740 1 -- 172.21.15.78:0/2142077870 wait complete. 2024-09-16T01:19:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.781+0000 7f914d5fc740 20 client.25724 trim_cache size 0 max 16384 2024-09-16T01:19:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.801+0000 7f914d5fc740 1 Processor -- start 2024-09-16T01:19:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.801+0000 7f914d5fc740 1 -- start start 2024-09-16T01:19:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df125340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df127cb0 0x5640df13f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f91441c0890 con 0x5640df127cb0 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91441be160 con 0x5640df11caf0 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f914d5fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91441bbfd0 con 0x5640df11d410 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:33624/0 (socket says 172.21.15.78:33624) 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 -- 172.21.15.78:0/1228676926 learned_addr learned my addr 172.21.15.78:0/1228676926 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9148dfb640 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df127cb0 0x5640df13f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f91437fe640 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df125340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 -- 172.21.15.78:0/1228676926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 msgr2=0x5640df125340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df125340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 -- 172.21.15.78:0/1228676926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df127cb0 msgr2=0x5640df13f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df127cb0 0x5640df13f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.802+0000 7f9143fff640 1 -- 172.21.15.78:0/1228676926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640df11d240 con 0x5640df11caf0 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.803+0000 7f9143fff640 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9138098b30 tx=0x7f91380d2040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c00ac6ef094743 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.803+0000 7f9129ffb640 1 -- 172.21.15.78:0/1228676926 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913809c160 con 0x5640df11caf0 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.803+0000 7f9129ffb640 1 -- 172.21.15.78:0/1228676926 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91380a0510 con 0x5640df11caf0 2024-09-16T01:19:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.803+0000 7f9129ffb640 1 -- 172.21.15.78:0/1228676926 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380bc0b0 con 0x5640df11caf0 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- 172.21.15.78:0/1228676926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 msgr2=0x5640df123340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9138098b30 tx=0x7f91380d2040 comp rx=0 tx=0).stop 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- 172.21.15.78:0/1228676926 shutdown_connections 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df127cb0 0x5640df13f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df125340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 --2- 172.21.15.78:0/1228676926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df11caf0 0x5640df123340 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- 172.21.15.78:0/1228676926 >> 172.21.15.78:0/1228676926 conn(0x5640df119060 msgr2=0x5640df126b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- 172.21.15.78:0/1228676926 shutdown_connections 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- 172.21.15.78:0/1228676926 wait complete. 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 -- start start 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.804+0000 7f914d5fc740 1 Processor -- start 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df130260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df121cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f91441bc620 con 0x5640df11caf0 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91441bda90 con 0x5640df127cb0 2024-09-16T01:19:03.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91441bccf0 con 0x5640df11d410 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f914d5fc740 10 client.0 _ll_register_callbacks cb 0x7f913401e050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f9143fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f9143fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57650/0 (socket says 172.21.15.78:57650) 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f9143fff640 1 -- 172.21.15.78:0/2742373194 learned_addr learned my addr 172.21.15.78:0/2742373194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.805+0000 7f9148dfb640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df121cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 -- 172.21.15.78:0/2742373194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 msgr2=0x5640df130260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df130260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 -- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 msgr2=0x5640df121cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df121cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 -- 172.21.15.78:0/2742373194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640df122230 con 0x5640df11caf0 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.806+0000 7f9143fff640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f913809c0f0 tx=0x7f91380d3940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b410b2a89854d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:03.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f91237fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380a4dd0 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9525+0+0 (secure 0 0 0) 0x7f913800fb30 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91380a0510 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f91380bc0e0 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f914d5fc740 10 client.16320 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.807+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640df1287e0 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91380a0510 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f91380a00d0 con 0x5640df11caf0 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 10 client.16320 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 10 client.16320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 20 client.16320 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df1a4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:03.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 4 client.16320 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:19:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.808+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640df088d70 con 0x5640df137b80 2024-09-16T01:19:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.809+0000 7f91437fe640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df1a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.810+0000 7f91437fe640 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df1a4670 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72895936e45caca1 server_cookie=189481cb195b0f7 in_seq=0 out_seq=0 2024-09-16T01:19:03.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.810+0000 7f91237fe640 10 client.16320 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f91237fe640 1 -- 172.21.15.78:0/2742373194 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640df088d70 con 0x5640df137b80 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f91237fe640 10 client.16320 handle_command_reply: tid=0 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 client.16320 shutdown 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 msgr2=0x5640df12fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f913809c0f0 tx=0x7f91380d3940 comp rx=0 tx=0).stop 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 shutdown_connections 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5640df137b80 0x5640df1a4670 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5640df127cb0 0x5640df121cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f91237fe640 0 client.16320 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640df11d410 0x5640df130260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 --2- 172.21.15.78:0/2742373194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640df11caf0 0x5640df12fd20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 >> 172.21.15.78:0/2742373194 conn(0x5640df119060 msgr2=0x5640df1179a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 shutdown_connections 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 1 -- 172.21.15.78:0/2742373194 wait complete. 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.818+0000 7f914d5fc740 20 client.16320 trim_cache size 0 max 16384 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:19:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:19:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:19:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:19:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 msgr2=0x7f912c0a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 0x7f912c0a5a80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9134011a80 tx=0x7f9134000bc0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 msgr2=0x7f91440826c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f913800c920 tx=0x7f913800cde0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 shutdown_connections 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f912c0a35c0 0x7f912c0a5a80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f914407b740 0x7f914407bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f914410d4b0 0x7f9144082c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 --2- 172.21.15.78:0/2945733571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9144108360 0x7f91440826c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.819+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 >> 172.21.15.78:0/2945733571 conn(0x7f9144077ad0 msgr2=0x7f9144078e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.820+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 shutdown_connections 2024-09-16T01:19:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:03.820+0000 7f914a85e640 1 -- 172.21.15.78:0/2945733571 wait complete. 2024-09-16T01:19:05.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:05 smithi078 ceph-mon[30937]: pgmap v255: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 522 op/s 2024-09-16T01:19:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:05 smithi193 ceph-mon[35701]: pgmap v255: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 522 op/s 2024-09-16T01:19:05.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:05 smithi148 ceph-mon[35030]: pgmap v255: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.1 MiB/s wr, 522 op/s 2024-09-16T01:19:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 470.0 s, 558.6 tps, lat 196.842 ms stddev 264.137 2024-09-16T01:19:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:07 smithi078 ceph-mon[30937]: pgmap v256: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-09-16T01:19:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:07 smithi193 ceph-mon[35701]: pgmap v256: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-09-16T01:19:07.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:07 smithi148 ceph-mon[35030]: pgmap v256: 129 pgs: 129 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-09-16T01:19:09.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:09 smithi078 ceph-mon[30937]: pgmap v257: 129 pgs: 129 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.9 MiB/s wr, 1.05k op/s 2024-09-16T01:19:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:09 smithi078 ceph-mon[30937]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T01:19:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:09 smithi193 ceph-mon[35701]: pgmap v257: 129 pgs: 129 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.9 MiB/s wr, 1.05k op/s 2024-09-16T01:19:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:09 smithi193 ceph-mon[35701]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T01:19:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:09 smithi148 ceph-mon[35030]: pgmap v257: 129 pgs: 129 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.9 MiB/s wr, 1.05k op/s 2024-09-16T01:19:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:09 smithi148 ceph-mon[35030]: osdmap e99: 12 total, 12 up, 12 in 2024-09-16T01:19:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 475.0 s, 1039.2 tps, lat 96.952 ms stddev 136.186 2024-09-16T01:19:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:11 smithi078 ceph-mon[30937]: pgmap v259: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-09-16T01:19:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:11 smithi193 ceph-mon[35701]: pgmap v259: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-09-16T01:19:11.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:11 smithi148 ceph-mon[35030]: pgmap v259: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-09-16T01:19:13.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:13 smithi078 ceph-mon[30937]: pgmap v260: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:13 smithi193 ceph-mon[35701]: pgmap v260: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:13.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:13 smithi148 ceph-mon[35030]: pgmap v260: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:15 smithi078 ceph-mon[30937]: pgmap v261: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:15 smithi193 ceph-mon[35701]: pgmap v261: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:15.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:15 smithi148 ceph-mon[35030]: pgmap v261: 129 pgs: 5 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 112 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-09-16T01:19:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 480.0 s, 776.8 tps, lat 129.227 ms stddev 158.734 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 -- 172.21.15.78:0/218956290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c101370 msgr2=0x7f337c10d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 --2- 172.21.15.78:0/218956290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c101370 0x7f337c10d8a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3374009be0 tx=0x7f337402a6c0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 -- 172.21.15.78:0/218956290 shutdown_connections 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 --2- 172.21.15.78:0/218956290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c101370 0x7f337c10d8a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 --2- 172.21.15.78:0/218956290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 0x7f337c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 --2- 172.21.15.78:0/218956290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c1069e0 0x7f337c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 -- 172.21.15.78:0/218956290 >> 172.21.15.78:0/218956290 conn(0x7f337c0fc6f0 msgr2=0x7f337c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 -- 172.21.15.78:0/218956290 shutdown_connections 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.973+0000 7f3382b2c640 1 -- 172.21.15.78:0/218956290 wait complete. 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.974+0000 7f3382b2c640 1 Processor -- start 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.974+0000 7f3382b2c640 1 -- start start 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 0x7f337c073950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c101370 0x7f337c073e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f337c06c620 con 0x7f337c101370 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f337c06c4a0 con 0x7f337c1009d0 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f3382b2c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f337c06c7a0 con 0x7f337c1069e0 2024-09-16T01:19:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:53912/0 (socket says 172.21.15.78:53912) 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 -- 172.21.15.78:0/2784156319 learned_addr learned my addr 172.21.15.78:0/2784156319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33808c8640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 0x7f337c073950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f337bfff640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c101370 0x7f337c073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 -- 172.21.15.78:0/2784156319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 msgr2=0x7f337c073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 0x7f337c073950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 -- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c101370 msgr2=0x7f337c073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c101370 0x7f337c073e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.975+0000 7f33810c9640 1 -- 172.21.15.78:0/2784156319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c10d3f0 con 0x7f337c1069e0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f33810c9640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3374030c90 tx=0x7f3374032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f3379ffb640 1 -- 172.21.15.78:0/2784156319 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374041070 con 0x7f337c1069e0 2024-09-16T01:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f3379ffb640 1 -- 172.21.15.78:0/2784156319 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3374003e50 con 0x7f337c1069e0 2024-09-16T01:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f3379ffb640 1 -- 172.21.15.78:0/2784156319 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374036070 con 0x7f337c1069e0 2024-09-16T01:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c10d620 con 0x7f337c1069e0 2024-09-16T01:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.976+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c1ab400 con 0x7f337c1069e0 2024-09-16T01:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.980+0000 7f3379ffb640 1 -- 172.21.15.78:0/2784156319 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3374044070 con 0x7f337c1069e0 2024-09-16T01:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.981+0000 7f3379ffb640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 0x7f336c0e9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.981+0000 7f3379ffb640 1 -- 172.21.15.78:0/2784156319 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f33740f06d0 con 0x7f337c1069e0 2024-09-16T01:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.982+0000 7f33808c8640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 0x7f336c0e9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.983+0000 7f33808c8640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 0x7f336c0e9a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33700096c0 tx=0x7f3370009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.983+0000 7f33858c2740 1 Processor -- start 2024-09-16T01:19:16.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.983+0000 7f33858c2740 1 -- start start 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0ef410 0x56204a0f9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56204a0532f0 con 0x56204a0ef410 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562049ffb270 con 0x56204a0f9cb0 2024-09-16T01:19:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33858c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56204a04f2b0 con 0x56204a0eeaf0 2024-09-16T01:19:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33810c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33810c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:53590/0 (socket says 172.21.15.78:53590) 2024-09-16T01:19:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33810c9640 1 -- 172.21.15.78:0/1630436049 learned_addr learned my addr 172.21.15.78:0/1630436049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33808c8640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f337bfff640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0ef410 0x56204a0f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33808c8640 1 -- 172.21.15.78:0/1630436049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 msgr2=0x56204a0fc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.984+0000 7f33808c8640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f33808c8640 1 -- 172.21.15.78:0/1630436049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0ef410 msgr2=0x56204a0f9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f33808c8640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0ef410 0x56204a0f9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f33808c8640 1 -- 172.21.15.78:0/1630436049 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562049ffa340 con 0x56204a0eeaf0 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f33810c9640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f33808c8640 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0eeed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3370005fd0 tx=0x7f3370014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e80e4252623d0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f3366ffd640 1 -- 172.21.15.78:0/1630436049 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000f820 con 0x56204a0eeaf0 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f3366ffd640 1 -- 172.21.15.78:0/1630436049 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3370026400 con 0x56204a0eeaf0 2024-09-16T01:19:16.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.985+0000 7f3366ffd640 1 -- 172.21.15.78:0/1630436049 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000fea0 con 0x56204a0eeaf0 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 -- 172.21.15.78:0/1630436049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 msgr2=0x56204a0eeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0eeed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3370005fd0 tx=0x7f3370014000 comp rx=0 tx=0).stop 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 -- 172.21.15.78:0/1630436049 shutdown_connections 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0ef410 0x56204a0f9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 --2- 172.21.15.78:0/1630436049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0eeed0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 -- 172.21.15.78:0/1630436049 >> 172.21.15.78:0/1630436049 conn(0x56204a0eb060 msgr2=0x56204a0eb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 -- 172.21.15.78:0/1630436049 shutdown_connections 2024-09-16T01:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.986+0000 7f33858c2740 1 -- 172.21.15.78:0/1630436049 wait complete. 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 -- start start 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 Processor -- start 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56204a0fc8d0 con 0x56204a0eeaf0 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562049fdc290 con 0x56204a0f9cb0 2024-09-16T01:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56204a0fca50 con 0x56204a0ef410 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.987+0000 7f33858c2740 10 client.0 _ll_register_callbacks cb 0x56204a00d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33808c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33808c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37844/0 (socket says 172.21.15.78:37844) 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33808c8640 1 -- 172.21.15.78:0/3889916327 learned_addr learned my addr 172.21.15.78:0/3889916327 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f337bfff640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 -- 172.21.15.78:0/3889916327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 msgr2=0x56204a0f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 -- 172.21.15.78:0/3889916327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 msgr2=0x56204a0f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 -- 172.21.15.78:0/3889916327 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562049d66210 con 0x56204a0f9cb0 2024-09-16T01:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33808c8640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f337bfff640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f33810c9640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0f75b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f33740c4c50 tx=0x7f3374004180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60189d925fc0ed04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f335bfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.988+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33740d5060 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f33740bc770 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3370026400 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33740ede00 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 10 client.25742 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562049e32620 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3370026400 con 0x56204a0f9cb0 2024-09-16T01:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33740d5200 con 0x56204a0f9cb0 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 10 client.25742 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 20 client.25742 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.989+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a109bc0 0x56204a10bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.990+0000 7f33858c2740 4 client.25742 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.990+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56204a060a20 con 0x56204a109bc0 2024-09-16T01:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.990+0000 7f33808c8640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a109bc0 0x56204a10bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.991+0000 7f33808c8640 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a109bc0 0x56204a10bfd0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f0d6d4858d01f26 server_cookie=b7b5c6fd9f620768 in_seq=0 out_seq=0 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.991+0000 7f335bfff640 10 client.25742 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.991+0000 7f335bfff640 1 -- 172.21.15.78:0/3889916327 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56204a060a20 con 0x56204a109bc0 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.991+0000 7f335bfff640 10 client.25742 handle_command_reply: tid=0 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 client.25742 shutdown 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 msgr2=0x56204a0f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0f75b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f33740c4c50 tx=0x7f3374004180 comp rx=0 tx=0).stop 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 shutdown_connections 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a109bc0 0x56204a10bfd0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0f75b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f335bfff640 0 client.25742 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 --2- 172.21.15.78:0/3889916327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a0f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 >> 172.21.15.78:0/3889916327 conn(0x56204a0eb060 msgr2=0x56204a0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 shutdown_connections 2024-09-16T01:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 1 -- 172.21.15.78:0/3889916327 wait complete. 2024-09-16T01:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:16.993+0000 7f33858c2740 20 client.25742 trim_cache size 0 max 16384 2024-09-16T01:19:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 Processor -- start 2024-09-16T01:19:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 -- start start 2024-09-16T01:19:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0f1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 0x56204a1118e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33808c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33808c8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0f1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:53942/0 (socket says 172.21.15.78:53942) 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f337bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33810c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 0x56204a1118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f337bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:53598/0 (socket says 172.21.15.78:53598) 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33810c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 0x56204a1118e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37858/0 (socket says 172.21.15.78:37858) 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33808c8640 1 -- 172.21.15.78:0/2663900633 learned_addr learned my addr 172.21.15.78:0/2663900633 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b80b0 con 0x56204a0f9cb0 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b5980 con 0x56204a0ef410 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.023+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b37f0 con 0x56204a0eeaf0 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.024+0000 7f337bfff640 1 -- 172.21.15.78:0/2663900633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 msgr2=0x56204a0f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.024+0000 7f337bfff640 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0f1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.024+0000 7f337bfff640 1 -- 172.21.15.78:0/2663900633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 msgr2=0x56204a1118e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.024+0000 7f337bfff640 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 0x56204a1118e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.024+0000 7f337bfff640 1 -- 172.21.15.78:0/2663900633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56204a10c650 con 0x56204a0ef410 2024-09-16T01:19:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.028+0000 7f337bfff640 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x56204a0f8170 tx=0x7f3368005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0866421cc2fab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.035+0000 7f33667fc640 1 -- 172.21.15.78:0/2663900633 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368022070 con 0x56204a0ef410 2024-09-16T01:19:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.035+0000 7f33667fc640 1 -- 172.21.15.78:0/2663900633 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368004310 con 0x56204a0ef410 2024-09-16T01:19:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.035+0000 7f33667fc640 1 -- 172.21.15.78:0/2663900633 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680119c0 con 0x56204a0ef410 2024-09-16T01:19:17.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.036+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 msgr2=0x56204a0f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.036+0000 7f33858c2740 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x56204a0f8170 tx=0x7f3368005f90 comp rx=0 tx=0).stop 2024-09-16T01:19:17.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.037+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 shutdown_connections 2024-09-16T01:19:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.037+0000 7f33858c2740 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0f9cb0 0x56204a1118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.037+0000 7f33858c2740 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0ef410 0x56204a0f73e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.037+0000 7f33858c2740 1 --2- 172.21.15.78:0/2663900633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0eeaf0 0x56204a0f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.037+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 >> 172.21.15.78:0/2663900633 conn(0x56204a0eb060 msgr2=0x56204a0f8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:17.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.041+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 shutdown_connections 2024-09-16T01:19:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.046+0000 7f33858c2740 1 -- 172.21.15.78:0/2663900633 wait complete. 2024-09-16T01:19:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.046+0000 7f33858c2740 1 -- start start 2024-09-16T01:19:17.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.046+0000 7f33858c2740 1 Processor -- start 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a1022a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b3e40 con 0x56204a0eeaf0 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b52b0 con 0x56204a0f9cb0 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f337c1b4510 con 0x56204a0ef410 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f33858c2740 10 client.0 _ll_register_callbacks cb 0x7f33740bbe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f337bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f337bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:53958/0 (socket says 172.21.15.78:53958) 2024-09-16T01:19:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.047+0000 7f337bfff640 1 -- 172.21.15.78:0/3349829757 learned_addr learned my addr 172.21.15.78:0/3349829757 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f33810c9640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f337bfff640 1 -- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 msgr2=0x56204a0fdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f337bfff640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f337bfff640 1 -- 172.21.15.78:0/3349829757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 msgr2=0x56204a1022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f337bfff640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a1022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.051+0000 7f337bfff640 1 -- 172.21.15.78:0/3349829757 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56204a0ebcf0 con 0x56204a0ef410 2024-09-16T01:19:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.055+0000 7f337bfff640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3368003f90 tx=0x7f3368011fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2ff0d8b81e7f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.055+0000 7f335bfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:19:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.055+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336802e060 con 0x56204a0ef410 2024-09-16T01:19:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.055+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3368004310 con 0x56204a0ef410 2024-09-16T01:19:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.055+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9613+0+0 (secure 0 0 0) 0x7f33680035f0 con 0x56204a0ef410 2024-09-16T01:19:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.056+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3368035050 con 0x56204a0ef410 2024-09-16T01:19:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.056+0000 7f33858c2740 10 client.25774 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.056+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56204a0eae90 con 0x56204a0ef410 2024-09-16T01:19:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.057+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368004310 con 0x56204a0ef410 2024-09-16T01:19:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.057+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f33680044c0 con 0x56204a0ef410 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 10 client.25774 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 20 client.25774 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a174950 0x56204a176d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 4 client.25774 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:19:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56204a05ad70 con 0x56204a174950 2024-09-16T01:19:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.058+0000 7f33808c8640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a174950 0x56204a176d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.059+0000 7f335bfff640 10 client.25774 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.059+0000 7f33808c8640 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a174950 0x56204a176d70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411fbd4591088ffd server_cookie=f7ad2576323c47f2 in_seq=0 out_seq=0 2024-09-16T01:19:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.088+0000 7f335bfff640 1 -- 172.21.15.78:0/3349829757 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56204a05ad70 con 0x56204a174950 2024-09-16T01:19:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.088+0000 7f335bfff640 10 client.25774 handle_command_reply: tid=0 2024-09-16T01:19:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.088+0000 7f33858c2740 1 client.25774 shutdown 2024-09-16T01:19:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 msgr2=0x56204a0fd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3368003f90 tx=0x7f3368011fc0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 shutdown_connections 2024-09-16T01:19:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x56204a174950 0x56204a176d70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56204a0f9cb0 0x56204a0fdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56204a0ef410 0x56204a0fd610 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 --2- 172.21.15.78:0/3349829757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56204a0eeaf0 0x56204a1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 >> 172.21.15.78:0/3349829757 conn(0x56204a0eb060 msgr2=0x56204a0e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f335bfff640 0 client.25774 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 shutdown_connections 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 1 -- 172.21.15.78:0/3349829757 wait complete. 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.093+0000 7f33858c2740 20 client.25774 trim_cache size 0 max 16384 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:19:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:19:17.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:19:17.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 msgr2=0x7f336c0e9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 0x7f336c0e9a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33700096c0 tx=0x7f3370009260 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 msgr2=0x7f337c10cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3374030c90 tx=0x7f3374032f90 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 shutdown_connections 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f336c0e7570 0x7f336c0e9a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f337c1069e0 0x7f337c10cb30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f337c101370 0x7f337c073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 --2- 172.21.15.78:0/2784156319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f337c1009d0 0x7f337c073950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 >> 172.21.15.78:0/2784156319 conn(0x7f337c0fc6f0 msgr2=0x7f337c10dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 shutdown_connections 2024-09-16T01:19:17.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:17.095+0000 7f3382b2c640 1 -- 172.21.15.78:0/2784156319 wait complete. 2024-09-16T01:19:17.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:17 smithi078 ceph-mon[30937]: pgmap v262: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 5.7 MiB/s wr, 627 op/s 2024-09-16T01:19:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:17 smithi193 ceph-mon[35701]: pgmap v262: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 5.7 MiB/s wr, 627 op/s 2024-09-16T01:19:17.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:17 smithi148 ceph-mon[35030]: pgmap v262: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 5.7 MiB/s wr, 627 op/s 2024-09-16T01:19:19.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:19 smithi078 ceph-mon[30937]: pgmap v263: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-09-16T01:19:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:19 smithi193 ceph-mon[35701]: pgmap v263: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-09-16T01:19:19.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:19 smithi148 ceph-mon[35030]: pgmap v263: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.4 MiB/s wr, 560 op/s 2024-09-16T01:19:20.786 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T01:19:20.787 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-16T01:19:20.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:19:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 485.0 s, 795.8 tps, lat 122.358 ms stddev 187.198 2024-09-16T01:19:21.301 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:21 smithi193 ceph-mon[35701]: pgmap v264: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.7 MiB/s wr, 685 op/s 2024-09-16T01:19:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:21 smithi193 ceph-mon[35701]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T01:19:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:21 smithi078 ceph-mon[30937]: pgmap v264: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.7 MiB/s wr, 685 op/s 2024-09-16T01:19:21.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:21 smithi078 ceph-mon[30937]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T01:19:21.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:21 smithi148 ceph-mon[35030]: pgmap v264: 129 pgs: 129 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.7 MiB/s wr, 685 op/s 2024-09-16T01:19:21.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:21 smithi148 ceph-mon[35030]: osdmap e100: 12 total, 12 up, 12 in 2024-09-16T01:19:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 -- 172.21.15.78:0/3129299283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 msgr2=0x7fe18011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 --2- 172.21.15.78:0/3129299283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe18011d530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe178009ca0 tx=0x7fe17802a8f0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 -- 172.21.15.78:0/3129299283 shutdown_connections 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 --2- 172.21.15.78:0/3129299283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe18011d530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 --2- 172.21.15.78:0/3129299283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180109830 0x7fe180109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 --2- 172.21.15.78:0/3129299283 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180109450 0x7fe180110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.891+0000 7fe188323640 1 -- 172.21.15.78:0/3129299283 >> 172.21.15.78:0/3129299283 conn(0x7fe18006e140 msgr2=0x7fe18006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:21.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.906+0000 7fe188323640 1 -- 172.21.15.78:0/3129299283 shutdown_connections 2024-09-16T01:19:21.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.906+0000 7fe188323640 1 -- 172.21.15.78:0/3129299283 wait complete. 2024-09-16T01:19:21.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.907+0000 7fe188323640 1 Processor -- start 2024-09-16T01:19:21.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.909+0000 7fe188323640 1 -- start start 2024-09-16T01:19:21.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.909+0000 7fe188323640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe180118950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.909+0000 7fe188323640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1801119e0 0x7fe180111e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.909+0000 7fe188323640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 0x7fe180116480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.909+0000 7fe188323640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe180120e80 con 0x7fe1801119e0 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.910+0000 7fe188323640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe180120d00 con 0x7fe180112380 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.910+0000 7fe188323640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe180121000 con 0x7fe18011b140 2024-09-16T01:19:21.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.911+0000 7fe1868c0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 0x7fe180116480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.913+0000 7fe1860bf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe180118950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.913+0000 7fe1860bf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe180118950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:53966/0 (socket says 172.21.15.78:53966) 2024-09-16T01:19:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.913+0000 7fe1860bf640 1 -- 172.21.15.78:0/1474600166 learned_addr learned my addr 172.21.15.78:0/1474600166 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:21.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.919+0000 7fe1868c0640 1 -- 172.21.15.78:0/1474600166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 msgr2=0x7fe180118950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:21.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.919+0000 7fe1868c0640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe180118950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.919+0000 7fe1868c0640 1 -- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1801119e0 msgr2=0x7fe180111e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:21.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.919+0000 7fe1868c0640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1801119e0 0x7fe180111e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:21.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.919+0000 7fe1868c0640 1 -- 172.21.15.78:0/1474600166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1801169c0 con 0x7fe180112380 2024-09-16T01:19:21.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.923+0000 7fe1868c0640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 0x7fe180116480 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe178009ca0 tx=0x7fe178031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:21.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.928+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1780307f0 con 0x7fe180112380 2024-09-16T01:19:21.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.928+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17802d070 con 0x7fe180112380 2024-09-16T01:19:21.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.928+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe180116c50 con 0x7fe180112380 2024-09-16T01:19:21.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.928+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178004740 con 0x7fe180112380 2024-09-16T01:19:21.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.929+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe180117190 con 0x7fe180112380 2024-09-16T01:19:21.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.931+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe180073b00 con 0x7fe180112380 2024-09-16T01:19:21.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.933+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fe178030990 con 0x7fe180112380 2024-09-16T01:19:21.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.934+0000 7fe1777fe640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 0x7fe16c0a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:21.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.935+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fe1780f0fd0 con 0x7fe180112380 2024-09-16T01:19:21.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.935+0000 7fe1860bf640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 0x7fe16c0a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:21.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.937+0000 7fe1860bf640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 0x7fe16c0a5a40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe1700058c0 tx=0x7fe17000a250 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:21.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:21.938+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1780bd6f0 con 0x7fe180112380 2024-09-16T01:19:22.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.287+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe18010b400 con 0x7fe180112380 2024-09-16T01:19:22.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.293+0000 7fe1777fe640 1 -- 172.21.15.78:0/1474600166 <== mon.2 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fe1780bcff0 con 0x7fe180112380 2024-09-16T01:19:22.295 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:19:22.295 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:19:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:19:22.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.299+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 msgr2=0x7fe16c0a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.299+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 0x7fe16c0a5a40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe1700058c0 tx=0x7fe17000a250 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.299+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 msgr2=0x7fe180116480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.299+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 0x7fe180116480 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe178009ca0 tx=0x7fe178031a20 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 shutdown_connections 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe16c0a3580 0x7fe16c0a5a40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe180112380 0x7fe180116480 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1801119e0 0x7fe180111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 --2- 172.21.15.78:0/1474600166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18011b140 0x7fe180118950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.301+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 >> 172.21.15.78:0/1474600166 conn(0x7fe18006e140 msgr2=0x7fe18011c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:22.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.307+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 shutdown_connections 2024-09-16T01:19:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:22.310+0000 7fe188323640 1 -- 172.21.15.78:0/1474600166 wait complete. 2024-09-16T01:19:23.234 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-16T01:19:23.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-16T01:19:23.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:23 smithi078 ceph-mon[30937]: pgmap v266: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.6 MiB/s wr, 837 op/s 2024-09-16T01:19:23.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:23 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1474600166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:23 smithi193 ceph-mon[35701]: pgmap v266: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.6 MiB/s wr, 837 op/s 2024-09-16T01:19:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:23 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1474600166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:23.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:23 smithi148 ceph-mon[35030]: pgmap v266: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.6 MiB/s wr, 837 op/s 2024-09-16T01:19:23.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:23 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1474600166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:23.665 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:24.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.306+0000 7f93a9418640 1 -- 172.21.15.78:0/2079308801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a41019c0 msgr2=0x7f93a4101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.306+0000 7f93a9418640 1 --2- 172.21.15.78:0/2079308801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a41019c0 0x7f93a4101e20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9398009930 tx=0x7f939802a420 comp rx=0 tx=0).stop 2024-09-16T01:19:24.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 -- 172.21.15.78:0/2079308801 shutdown_connections 2024-09-16T01:19:24.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 --2- 172.21.15.78:0/2079308801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a4102360 0x7f93a410e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 --2- 172.21.15.78:0/2079308801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a41019c0 0x7f93a4101e20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 --2- 172.21.15.78:0/2079308801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41079c0 0x7f93a4107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 -- 172.21.15.78:0/2079308801 >> 172.21.15.78:0/2079308801 conn(0x7f93a40fd6c0 msgr2=0x7f93a40ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 -- 172.21.15.78:0/2079308801 shutdown_connections 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.307+0000 7f93a9418640 1 -- 172.21.15.78:0/2079308801 wait complete. 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.308+0000 7f93a9418640 1 Processor -- start 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.308+0000 7f93a9418640 1 -- start start 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.308+0000 7f93a9418640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 0x7f93a4198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a9418640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 0x7f93a4198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 0x7f93a4198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58524/0 (socket says 172.21.15.78:58524) 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51982/0 (socket says 172.21.15.78:51982) 2024-09-16T01:19:24.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a37fe640 1 -- 172.21.15.78:0/452935848 learned_addr learned my addr 172.21.15.78:0/452935848 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a41079c0 0x7f93a419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4111dd0 con 0x7f93a4102360 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4111c50 con 0x7f93a41079c0 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a8c17640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a41079c0 0x7f93a419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.309+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f93a4111f50 con 0x7f93a41019c0 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 -- 172.21.15.78:0/452935848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 msgr2=0x7f93a4198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 0x7f93a4198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 -- 172.21.15.78:0/452935848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a41079c0 msgr2=0x7f93a419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a41079c0 0x7f93a419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a419d0e0 con 0x7f93a4102360 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a37fe640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f939802a940 tx=0x7f9398030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a17fa640 1 -- 172.21.15.78:0/452935848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939803f070 con 0x7f93a4102360 2024-09-16T01:19:24.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a419d370 con 0x7f93a4102360 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a41a4cb0 con 0x7f93a4102360 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.310+0000 7f93a17fa640 1 -- 172.21.15.78:0/452935848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9398004460 con 0x7f93a4102360 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.311+0000 7f93a17fa640 1 -- 172.21.15.78:0/452935848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398049060 con 0x7f93a4102360 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.314+0000 7f93aca53740 1 Processor -- start 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.315+0000 7f93aca53740 1 -- start start 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 0x55f1ff0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a17fa640 1 -- 172.21.15.78:0/452935848 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f9398042070 con 0x7f93a4102360 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f1ff027160 con 0x55f1ff0d1c60 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f1fefce720 con 0x55f1ff0ccc20 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93aca53740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f1ff023120 con 0x55f1ff0d2580 2024-09-16T01:19:24.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a8c17640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a8c17640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51998/0 (socket says 172.21.15.78:51998) 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a8c17640 1 -- 172.21.15.78:0/1484698300 learned_addr learned my addr 172.21.15.78:0/1484698300 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 0x55f1ff0cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 -- 172.21.15.78:0/1484698300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0ccc20 msgr2=0x55f1ff0cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 -- 172.21.15.78:0/1484698300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 msgr2=0x55f1ff0d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.316+0000 7f93a3fff640 1 -- 172.21.15.78:0/1484698300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1fed98ba0 con 0x55f1ff0d2580 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 0x55f1ff0cc5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f939c009fa0 tx=0x7f939c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc75e7526f777790 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f938a7fc640 1 -- 172.21.15.78:0/1484698300 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c003c30 con 0x55f1ff0d2580 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f938a7fc640 1 -- 172.21.15.78:0/1484698300 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c003dd0 con 0x55f1ff0d2580 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f93a17fa640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 0x7f938c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f938a7fc640 1 -- 172.21.15.78:0/1484698300 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c003c30 con 0x55f1ff0d2580 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f93a17fa640 1 -- 172.21.15.78:0/452935848 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f93980f0180 con 0x7f93a4102360 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.317+0000 7f93a8c17640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 0x7f938c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 -- 172.21.15.78:0/1484698300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 msgr2=0x55f1ff0cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 0x55f1ff0cc5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f939c009fa0 tx=0x7f939c0118d0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93a8c17640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 0x7f938c0a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9394009ee0 tx=0x7f9394005f90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 -- 172.21.15.78:0/1484698300 shutdown_connections 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0d2580 0x55f1ff0cc5b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 --2- 172.21.15.78:0/1484698300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 -- 172.21.15.78:0/1484698300 >> 172.21.15.78:0/1484698300 conn(0x55f1ff0bea60 msgr2=0x55f1ff0bee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 -- 172.21.15.78:0/1484698300 shutdown_connections 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.318+0000 7f93aca53740 1 -- 172.21.15.78:0/1484698300 wait complete. 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 -- start start 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 Processor -- start 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0d6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f1ff0d8ff0 con 0x55f1ff0d1c60 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f1fefaf900 con 0x55f1ff0d2580 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93aca53740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f1ff0d9170 con 0x55f1ff0ccc20 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.319+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0d6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52004/0 (socket says 172.21.15.78:52004) 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a3fff640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a37fe640 1 -- 172.21.15.78:0/144100737 learned_addr learned my addr 172.21.15.78:0/144100737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93aca53740 10 client.0 _ll_register_callbacks cb 0x55f1fefad780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 -- 172.21.15.78:0/144100737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 msgr2=0x55f1ff0d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0d59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 -- 172.21.15.78:0/144100737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 msgr2=0x55f1ff0d5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 -- 172.21.15.78:0/144100737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1fefefc50 con 0x55f1ff0d2580 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a3fff640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0d59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a37fe640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:19:24.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93a8c17640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0d6430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f939401e970 tx=0x7f9394004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddba8b378d2d5a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.320+0000 7f93837fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394005020 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f939c003dd0 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f9394018030 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9394034050 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93aca53740 10 client.25763 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.321+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f1ff0e4d80 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939c003dd0 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9394029420 con 0x55f1ff0d2580 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 10 client.25763 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 10 client.25763 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 20 client.25763 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:24.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0e61e0 0x55f1ff0e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 4 client.25763 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.322+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1ff034890 con 0x55f1ff0e61e0 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.326+0000 7f93a3fff640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0e61e0 0x55f1ff0e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.343+0000 7f93a3fff640 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0e61e0 0x55f1ff0e86a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4deefcf3d2875112 server_cookie=8dbd5a35dde8128b in_seq=0 out_seq=0 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.343+0000 7f93837fe640 10 client.25763 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.348+0000 7f93837fe640 1 -- 172.21.15.78:0/144100737 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1ff034890 con 0x55f1ff0e61e0 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.348+0000 7f93837fe640 10 client.25763 handle_command_reply: tid=0 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.348+0000 7f93aca53740 1 client.25763 shutdown 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.348+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 msgr2=0x55f1ff0d6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.348+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0d6430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f939401e970 tx=0x7f9394004730 comp rx=0 tx=0).stop 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 shutdown_connections 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0e61e0 0x55f1ff0e86a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0d6430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0d5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 --2- 172.21.15.78:0/144100737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0d59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 >> 172.21.15.78:0/144100737 conn(0x55f1ff0bea60 msgr2=0x55f1ff0be050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.349+0000 7f93837fe640 0 client.25763 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.352+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 shutdown_connections 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.355+0000 7f93aca53740 1 -- 172.21.15.78:0/144100737 wait complete. 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.355+0000 7f93aca53740 20 client.25763 trim_cache size 0 max 16384 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.390+0000 7f93aca53740 1 Processor -- start 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.390+0000 7f93aca53740 1 -- start start 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.390+0000 7f93aca53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.390+0000 7f93aca53740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0edf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.390+0000 7f93aca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0edf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52016/0 (socket says 172.21.15.78:52016) 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a37fe640 1 -- 172.21.15.78:0/115580633 learned_addr learned my addr 172.21.15.78:0/115580633 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9390012910 con 0x55f1ff0d1c60 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f93900101e0 con 0x55f1ff0d2580 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f939000e050 con 0x55f1ff0ccc20 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 -- 172.21.15.78:0/115580633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 msgr2=0x55f1ff0cc5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cc5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 -- 172.21.15.78:0/115580633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 msgr2=0x55f1ff0edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.391+0000 7f93a8c17640 1 -- 172.21.15.78:0/115580633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ff0ba2d0 con 0x55f1ff0d2580 2024-09-16T01:19:24.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.392+0000 7f93a8c17640 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0ee4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9394013010 tx=0x7f9394009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=344967d72d5bc214 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.392+0000 7f9389ffb640 1 -- 172.21.15.78:0/115580633 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939404d020 con 0x55f1ff0d2580 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.392+0000 7f9389ffb640 1 -- 172.21.15.78:0/115580633 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9394029420 con 0x55f1ff0d2580 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 msgr2=0x55f1ff0ee4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0ee4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9394013010 tx=0x7f9394009cb0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 shutdown_connections 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d2580 0x55f1ff0ee4b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0d1c60 0x55f1ff0edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 --2- 172.21.15.78:0/115580633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff0ccc20 0x55f1ff0cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.393+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 >> 172.21.15.78:0/115580633 conn(0x55f1ff0bea60 msgr2=0x55f1ff0be050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.397+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 shutdown_connections 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.397+0000 7f93aca53740 1 -- 172.21.15.78:0/115580633 wait complete. 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.397+0000 7f93aca53740 1 -- start start 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.397+0000 7f93aca53740 1 Processor -- start 2024-09-16T01:19:24.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d1c60 0x55f1ff14b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0dd450 0x55f1ff0dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f939000e6a0 con 0x55f1ff0dd450 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f939000fb10 con 0x55f1ff0d1c60 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f939000ed70 con 0x55f1ff14b5f0 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93aca53740 10 client.0 _ll_register_callbacks cb 0x7f9394012f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93a37fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58552/0 (socket says 172.21.15.78:58552) 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93a3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d1c60 0x55f1ff14b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.398+0000 7f93a37fe640 1 -- 172.21.15.78:0/1940979389 learned_addr learned my addr 172.21.15.78:0/1940979389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a8c17640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0dd450 0x55f1ff0dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 -- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d1c60 msgr2=0x55f1ff14b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d1c60 0x55f1ff14b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 -- 172.21.15.78:0/1940979389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0dd450 msgr2=0x55f1ff0dd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0dd450 0x55f1ff0dd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 -- 172.21.15.78:0/1940979389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1ff0ddec0 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93a37fe640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9398004650 tx=0x7f93980f38a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ad38ec36336a25d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93837fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.399+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93980bbef0 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.400+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1ff02ebe0 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.400+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f93980bc380 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.400+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f93980bba80 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.401+0000 7f93aca53740 10 client.25795 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.401+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f1ff0c6db0 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.402+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f1ff02ebe0 con 0x55f1ff14b5f0 2024-09-16T01:19:24.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.402+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f93980c4850 con 0x55f1ff14b5f0 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 10 client.25795 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 10 client.25795 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 20 client.25795 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0c75d0 0x55f1ff0c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 4 client.25795 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.403+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x55f1fefaaf40 con 0x55f1ff0c75d0 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.408+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0c75d0 0x55f1ff0c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.412+0000 7f93837fe640 10 client.25795 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:24.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.412+0000 7f93a3fff640 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0c75d0 0x55f1ff0c99f0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b046c724ac60000e server_cookie=d8540480869a7d08 in_seq=0 out_seq=0 2024-09-16T01:19:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93837fe640 1 -- 172.21.15.78:0/1940979389 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f1fefaaf40 con 0x55f1ff0c75d0 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93837fe640 10 client.25795 handle_command_reply: tid=0 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 client.25795 shutdown 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 msgr2=0x55f1ff14ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9398004650 tx=0x7f93980f38a0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 shutdown_connections 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f1ff0c75d0 0x55f1ff0c99f0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1ff0dd450 0x55f1ff0dd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1ff14b5f0 0x55f1ff14ba70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93837fe640 0 client.25795 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 --2- 172.21.15.78:0/1940979389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1ff0d1c60 0x55f1ff14b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 >> 172.21.15.78:0/1940979389 conn(0x55f1ff0bea60 msgr2=0x55f1ff0d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 shutdown_connections 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 1 -- 172.21.15.78:0/1940979389 wait complete. 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.683+0000 7f93aca53740 20 client.25795 trim_cache size 0 max 16384 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 30, 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:19:24.687 INFO:teuthology.orchestra.run.smithi078.stdout: "d69f434f": { 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 30, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 142.3, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:19:24.688 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:19:24.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 msgr2=0x7f938c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 0x7f938c0a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9394009ee0 tx=0x7f9394005f90 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 msgr2=0x7f93a4198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f939802a940 tx=0x7f9398030650 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 shutdown_connections 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f938c0a33b0 0x7f938c0a5870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a41079c0 0x7f93a419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a4102360 0x7f93a4198670 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 --2- 172.21.15.78:0/452935848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93a41019c0 0x7f93a4198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 >> 172.21.15.78:0/452935848 conn(0x7f93a40fd6c0 msgr2=0x7f93a40fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 shutdown_connections 2024-09-16T01:19:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:24.684+0000 7f93a9418640 1 -- 172.21.15.78:0/452935848 wait complete. 2024-09-16T01:19:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:25 smithi193 ceph-mon[35701]: pgmap v267: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 833 op/s 2024-09-16T01:19:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:25 smithi148 ceph-mon[35030]: pgmap v267: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 833 op/s 2024-09-16T01:19:25.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:25 smithi078 ceph-mon[30937]: pgmap v267: 129 pgs: 129 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.4 MiB/s wr, 833 op/s 2024-09-16T01:19:25.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 490.0 s, 603.1 tps, lat 116.689 ms stddev 138.482 2024-09-16T01:19:26.583 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd69f434f', quiesce took 5.8 seconds. Will release after: 16.55 2024-09-16T01:19:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:27 smithi193 ceph-mon[35701]: pgmap v268: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.0 MiB/s wr, 963 op/s 2024-09-16T01:19:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:27 smithi148 ceph-mon[35030]: pgmap v268: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.0 MiB/s wr, 963 op/s 2024-09-16T01:19:27.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:27 smithi078 ceph-mon[30937]: pgmap v268: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 9.0 MiB/s wr, 963 op/s 2024-09-16T01:19:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:29 smithi193 ceph-mon[35701]: pgmap v269: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.4 MiB/s wr, 931 op/s 2024-09-16T01:19:29.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:29 smithi148 ceph-mon[35030]: pgmap v269: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.4 MiB/s wr, 931 op/s 2024-09-16T01:19:29.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:29 smithi078 ceph-mon[30937]: pgmap v269: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.4 MiB/s wr, 931 op/s 2024-09-16T01:19:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 -- 172.21.15.78:0/4149922588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 msgr2=0x7f916c106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 --2- 172.21.15.78:0/4149922588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c106da0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f915c009930 tx=0x7f915c02a420 comp rx=0 tx=0).stop 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 -- 172.21.15.78:0/4149922588 shutdown_connections 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 --2- 172.21.15.78:0/4149922588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c101350 0x7f916c10d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 --2- 172.21.15.78:0/4149922588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c1009b0 0x7f916c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 --2- 172.21.15.78:0/4149922588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c106da0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 -- 172.21.15.78:0/4149922588 >> 172.21.15.78:0/4149922588 conn(0x7f916c077990 msgr2=0x7f916c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 -- 172.21.15.78:0/4149922588 shutdown_connections 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.192+0000 7f9171598640 1 -- 172.21.15.78:0/4149922588 wait complete. 2024-09-16T01:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 Processor -- start 2024-09-16T01:19:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 -- start start 2024-09-16T01:19:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 0x7f916c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 0x7f916c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f916c110e00 con 0x7f916c1069c0 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f916c110c80 con 0x7f916c1009b0 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f9171598640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f916c110f80 con 0x7f916c101350 2024-09-16T01:19:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52036/0 (socket says 172.21.15.78:52036) 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 -- 172.21.15.78:0/215824215 learned_addr learned my addr 172.21.15.78:0/215824215 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916a7fc640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 0x7f916c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916affd640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 0x7f916c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 -- 172.21.15.78:0/215824215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 msgr2=0x7f916c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 0x7f916c199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 -- 172.21.15.78:0/215824215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 msgr2=0x7f916c199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 0x7f916c199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916b7fe640 1 -- 172.21.15.78:0/215824215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916c1a47c0 con 0x7f916c1069c0 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916a7fc640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 0x7f916c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.193+0000 7f916affd640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 0x7f916c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f916b7fe640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f916400cad0 tx=0x7f916400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f9153fff640 1 -- 172.21.15.78:0/215824215 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916400ae20 con 0x7f916c1069c0 2024-09-16T01:19:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f9153fff640 1 -- 172.21.15.78:0/215824215 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9164005020 con 0x7f916c1069c0 2024-09-16T01:19:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916c1a4990 con 0x7f916c1069c0 2024-09-16T01:19:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f9153fff640 1 -- 172.21.15.78:0/215824215 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916400f6f0 con 0x7f916c1069c0 2024-09-16T01:19:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.194+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916c1a4e60 con 0x7f916c1069c0 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.196+0000 7f9153fff640 1 -- 172.21.15.78:0/215824215 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f91640051d0 con 0x7f916c1069c0 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.196+0000 7f9174410740 1 Processor -- start 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.196+0000 7f9174410740 1 -- start start 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb8a410 0x55a77bb94770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb970a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a77baee2f0 con 0x55a77bb89af0 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a77ba96270 con 0x55a77bb8a410 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9174410740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a77baea2b0 con 0x55a77bb94cb0 2024-09-16T01:19:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9153fff640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 0x7f91540a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52052/0 (socket says 172.21.15.78:52052) 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 -- 172.21.15.78:0/3480396110 learned_addr learned my addr 172.21.15.78:0/3480396110 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916affd640 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb8a410 0x55a77bb94770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916b7fe640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 0x7f91540a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 -- 172.21.15.78:0/3480396110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 msgr2=0x55a77bb970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 -- 172.21.15.78:0/3480396110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb8a410 msgr2=0x55a77bb94770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb8a410 0x55a77bb94770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 -- 172.21.15.78:0/3480396110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77ba95340 con 0x55a77bb89af0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f9153fff640 1 -- 172.21.15.78:0/215824215 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f91640d01b0 con 0x7f916c1069c0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f916000be80 tx=0x7f916000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfa4edcb6b7ab53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f916b7fe640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 0x7f91540a5920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91640cd5c0 tx=0x7f916406ad90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.197+0000 7f91517fa640 1 -- 172.21.15.78:0/3480396110 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160019030 con 0x55a77bb89af0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f91517fa640 1 -- 172.21.15.78:0/3480396110 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9160011a20 con 0x55a77bb89af0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f91517fa640 1 -- 172.21.15.78:0/3480396110 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916001d890 con 0x55a77bb89af0 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 -- 172.21.15.78:0/3480396110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 msgr2=0x55a77bb89ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f916000be80 tx=0x7f916000b730 comp rx=0 tx=0).stop 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 -- 172.21.15.78:0/3480396110 shutdown_connections 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb8a410 0x55a77bb94770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 --2- 172.21.15.78:0/3480396110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb89af0 0x55a77bb89ed0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 -- 172.21.15.78:0/3480396110 >> 172.21.15.78:0/3480396110 conn(0x55a77bb86060 msgr2=0x55a77bb86440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 -- 172.21.15.78:0/3480396110 shutdown_connections 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.198+0000 7f9174410740 1 -- 172.21.15.78:0/3480396110 wait complete. 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 -- start start 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 Processor -- start 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb91b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb925a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a77bb97880 con 0x55a77bb8a410 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a77ba77290 con 0x55a77bb89af0 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a77bb97a00 con 0x55a77bb94cb0 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f9174410740 10 client.0 _ll_register_callbacks cb 0x55a77baa8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52058/0 (socket says 172.21.15.78:52058) 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 -- 172.21.15.78:0/3808890935 learned_addr learned my addr 172.21.15.78:0/3808890935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb91b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 -- 172.21.15.78:0/3808890935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 msgr2=0x55a77bb925a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb925a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 -- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 msgr2=0x55a77bb91b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb91b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.199+0000 7f916affd640 1 -- 172.21.15.78:0/3808890935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77b801210 con 0x55a77bb8a410 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f916affd640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f915c033000 tx=0x7f915c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2a3fba4cd6b13b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c035040 con 0x55a77bb8a410 2024-09-16T01:19:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f915c004250 con 0x55a77bb8a410 2024-09-16T01:19:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9160011a20 con 0x55a77bb8a410 2024-09-16T01:19:30.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f915c04c050 con 0x55a77bb8a410 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 10 client.16365 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a77b8cd620 con 0x55a77bb8a410 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9160011a20 con 0x55a77bb8a410 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f915c004a40 con 0x55a77bb8a410 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 10 client.16365 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:30.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 20 client.16365 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bba4b80 0x55a77bba6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 4 client.16365 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.200+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77bafba20 con 0x55a77bba4b80 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.201+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bba4b80 0x55a77bba6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.201+0000 7f914eff5640 10 client.16365 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.201+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bba4b80 0x55a77bba6fd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1526430eb18cc3e server_cookie=c12576c6809f3f7d in_seq=0 out_seq=0 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f914eff5640 1 -- 172.21.15.78:0/3808890935 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a77bafba20 con 0x55a77bba4b80 2024-09-16T01:19:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f914eff5640 10 client.16365 handle_command_reply: tid=0 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 client.16365 shutdown 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 msgr2=0x55a77bb92060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f915c033000 tx=0x7f915c0317e0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 shutdown_connections 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bba4b80 0x55a77bba6fd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb94cb0 0x55a77bb925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb92060 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f914eff5640 0 client.16365 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 --2- 172.21.15.78:0/3808890935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb91b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 >> 172.21.15.78:0/3808890935 conn(0x55a77bb86060 msgr2=0x55a77bb8bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 shutdown_connections 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 1 -- 172.21.15.78:0/3808890935 wait complete. 2024-09-16T01:19:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.202+0000 7f9174410740 20 client.16365 trim_cache size 0 max 16384 2024-09-16T01:19:30.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 Processor -- start 2024-09-16T01:19:30.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 -- start start 2024-09-16T01:19:30.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb8a410 0x55a77bbac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1b1a10 con 0x55a77bb94cb0 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1af2e0 con 0x55a77bb89af0 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f9174410740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1ad150 con 0x55a77bb8a410 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.216+0000 7f916a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52072/0 (socket says 172.21.15.78:52072) 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 -- 172.21.15.78:0/1744479417 learned_addr learned my addr 172.21.15.78:0/1744479417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916affd640 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb8a410 0x55a77bbac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 -- 172.21.15.78:0/1744479417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb8a410 msgr2=0x55a77bbac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb8a410 0x55a77bbac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 -- 172.21.15.78:0/1744479417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 msgr2=0x55a77bb923d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 -- 172.21.15.78:0/1744479417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77bb85e90 con 0x55a77bb94cb0 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f916a7fc640 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f916001d540 tx=0x7f916000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc4f04221e143fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f9150ff9640 1 -- 172.21.15.78:0/1744479417 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160028040 con 0x55a77bb94cb0 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f9150ff9640 1 -- 172.21.15.78:0/1744479417 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916000b8d0 con 0x55a77bb94cb0 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.217+0000 7f9150ff9640 1 -- 172.21.15.78:0/1744479417 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160019040 con 0x55a77bb94cb0 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- 172.21.15.78:0/1744479417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 msgr2=0x55a77bbace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f916001d540 tx=0x7f916000b860 comp rx=0 tx=0).stop 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- 172.21.15.78:0/1744479417 shutdown_connections 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb94cb0 0x55a77bbace20 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb8a410 0x55a77bbac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- 172.21.15.78:0/1744479417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb89af0 0x55a77bb923d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- 172.21.15.78:0/1744479417 >> 172.21.15.78:0/1744479417 conn(0x55a77bb86060 msgr2=0x55a77bb93ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- 172.21.15.78:0/1744479417 shutdown_connections 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- 172.21.15.78:0/1744479417 wait complete. 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- start start 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 Processor -- start 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb89af0 0x55a77bb9d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 0x55a77bb98c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1ad7a0 con 0x55a77bb8a410 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1aec10 con 0x55a77bb94cb0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f916c1ade70 con 0x55a77bb89af0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f9174410740 10 client.0 _ll_register_callbacks cb 0x7f915c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f916affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f916affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52086/0 (socket says 172.21.15.78:52086) 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f916affd640 1 -- 172.21.15.78:0/3555484462 learned_addr learned my addr 172.21.15.78:0/3555484462 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.218+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb89af0 0x55a77bb9d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 0x55a77bb98c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 -- 172.21.15.78:0/3555484462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb89af0 msgr2=0x55a77bb9d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb89af0 0x55a77bb9d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 -- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 msgr2=0x55a77bb98c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:30.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 0x55a77bb98c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 -- 172.21.15.78:0/3555484462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77bb812d0 con 0x55a77bb8a410 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 0x55a77bb98c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f916affd640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f915c031550 tx=0x7f915c032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec664aceadf08d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c054070 con 0x55a77bb8a410 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f915c0048c0 con 0x55a77bb8a410 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a77baf5d70 con 0x55a77bb8a410 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f915c062050 con 0x55a77bb8a410 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f9174410740 10 client.16371 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:30.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a77bb87110 con 0x55a77bb8a410 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a77baf5d70 con 0x55a77bb8a410 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f915c0595f0 con 0x55a77bb8a410 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f9174410740 10 client.16371 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.219+0000 7f9174410740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f9174410740 20 client.16371 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bc0fc40 0x55a77bc12060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f9174410740 4 client.16371 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a77ba72900 con 0x55a77bc0fc40 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bc0fc40 0x55a77bc12060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f914eff5640 10 client.16371 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:30.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.220+0000 7f916a7fc640 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bc0fc40 0x55a77bc12060 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6f32f05335ee57a server_cookie=f1fc2df37e6cad3d in_seq=0 out_seq=0 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f914eff5640 1 -- 172.21.15.78:0/3555484462 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a77ba72900 con 0x55a77bc0fc40 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f914eff5640 10 client.16371 handle_command_reply: tid=0 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 client.16371 shutdown 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 msgr2=0x55a77bb98730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f915c031550 tx=0x7f915c032eb0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 shutdown_connections 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a77bc0fc40 0x55a77bc12060 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a77bb94cb0 0x55a77bb98c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a77bb8a410 0x55a77bb98730 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 --2- 172.21.15.78:0/3555484462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a77bb89af0 0x55a77bb9d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f914eff5640 0 client.16371 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 >> 172.21.15.78:0/3555484462 conn(0x55a77bb86060 msgr2=0x55a77bb8b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 shutdown_connections 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.228+0000 7f9174410740 1 -- 172.21.15.78:0/3555484462 wait complete. 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9174410740 20 client.16371 trim_cache size 0 max 16384 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:19:30.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:19:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 msgr2=0x7f91540a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 0x7f91540a5920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91640cd5c0 tx=0x7f916406ad90 comp rx=0 tx=0).stop 2024-09-16T01:19:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 msgr2=0x7f916c1a4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:30.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f916400cad0 tx=0x7f916400cf90 comp rx=0 tx=0).stop 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 shutdown_connections 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f91540a3460 0x7f91540a5920 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c1069c0 0x7f916c1a4280 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f916c101350 0x7f916c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 --2- 172.21.15.78:0/215824215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c1009b0 0x7f916c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 >> 172.21.15.78:0/215824215 conn(0x7f916c077990 msgr2=0x7f916c103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 shutdown_connections 2024-09-16T01:19:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:30.229+0000 7f9171598640 1 -- 172.21.15.78:0/215824215 wait complete. 2024-09-16T01:19:30.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 495.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:19:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:31 smithi193 ceph-mon[35701]: pgmap v270: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.5 MiB/s wr, 740 op/s 2024-09-16T01:19:31.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:31 smithi148 ceph-mon[35030]: pgmap v270: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.5 MiB/s wr, 740 op/s 2024-09-16T01:19:31.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:31 smithi078 ceph-mon[30937]: pgmap v270: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.5 MiB/s wr, 740 op/s 2024-09-16T01:19:33.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:33 smithi148 ceph-mon[35030]: pgmap v271: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.2 MiB/s wr, 469 op/s 2024-09-16T01:19:33.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:33 smithi078 ceph-mon[30937]: pgmap v271: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.2 MiB/s wr, 469 op/s 2024-09-16T01:19:33.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:33 smithi193 ceph-mon[35701]: pgmap v271: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 4.2 MiB/s wr, 469 op/s 2024-09-16T01:19:34.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:34 smithi148 ceph-mon[35030]: pgmap v272: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-09-16T01:19:34.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:34 smithi078 ceph-mon[30937]: pgmap v272: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-09-16T01:19:34.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:34 smithi193 ceph-mon[35701]: pgmap v272: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-09-16T01:19:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 500.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:19:36.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:36 smithi078 ceph-mon[30937]: pgmap v273: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-16T01:19:36.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:36 smithi193 ceph-mon[35701]: pgmap v273: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-16T01:19:36.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:36 smithi148 ceph-mon[35030]: pgmap v273: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-09-16T01:19:38.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:38 smithi078 ceph-mon[30937]: pgmap v274: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 337 KiB/s wr, 48 op/s 2024-09-16T01:19:38.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:38 smithi193 ceph-mon[35701]: pgmap v274: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 337 KiB/s wr, 48 op/s 2024-09-16T01:19:38.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:38 smithi148 ceph-mon[35030]: pgmap v274: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 337 KiB/s wr, 48 op/s 2024-09-16T01:19:40.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:40 smithi078 ceph-mon[30937]: pgmap v275: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 80 KiB/s wr, 18 op/s 2024-09-16T01:19:40.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:40 smithi193 ceph-mon[35701]: pgmap v275: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 80 KiB/s wr, 18 op/s 2024-09-16T01:19:40.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:40 smithi148 ceph-mon[35030]: pgmap v275: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 80 KiB/s wr, 18 op/s 2024-09-16T01:19:40.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 505.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:19:42.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:42 smithi078 ceph-mon[30937]: pgmap v276: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-09-16T01:19:42.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:42 smithi193 ceph-mon[35701]: pgmap v276: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-09-16T01:19:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:42 smithi148 ceph-mon[35030]: pgmap v276: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-09-16T01:19:43.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 -- 172.21.15.78:0/4247572615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4101350 msgr2=0x7f97b410d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 --2- 172.21.15.78:0/4247572615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4101350 0x7f97b410d880 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009cd0 tx=0x7f97ac02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 -- 172.21.15.78:0/4247572615 shutdown_connections 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 --2- 172.21.15.78:0/4247572615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4101350 0x7f97b410d880 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 --2- 172.21.15.78:0/4247572615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b41009b0 0x7f97b4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 --2- 172.21.15.78:0/4247572615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41069c0 0x7f97b4106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 -- 172.21.15.78:0/4247572615 >> 172.21.15.78:0/4247572615 conn(0x7f97b4077990 msgr2=0x7f97b40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 -- 172.21.15.78:0/4247572615 shutdown_connections 2024-09-16T01:19:43.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.322+0000 7f97bc781640 1 -- 172.21.15.78:0/4247572615 wait complete. 2024-09-16T01:19:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 Processor -- start 2024-09-16T01:19:43.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 -- start start 2024-09-16T01:19:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 0x7f97b4198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b4101350 0x7f97b4198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4110e00 con 0x7f97b41069c0 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4110c80 con 0x7f97b4101350 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bc781640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4110f80 con 0x7f97b41009b0 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bad1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bad1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41360/0 (socket says 172.21.15.78:41360) 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97bad1e640 1 -- 172.21.15.78:0/508118173 learned_addr learned my addr 172.21.15.78:0/508118173 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.323+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b4101350 0x7f97b4198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97ba51d640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 0x7f97b4198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 -- 172.21.15.78:0/508118173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 msgr2=0x7f97b4198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 0x7f97b4198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 -- 172.21.15.78:0/508118173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b4101350 msgr2=0x7f97b4198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b4101350 0x7f97b4198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 -- 172.21.15.78:0/508118173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b419d2f0 con 0x7f97b41069c0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97ba51d640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 0x7f97b4198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bad1e640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005cc0 tx=0x7f97ac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97ab7fe640 1 -- 172.21.15.78:0/508118173 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac004040 con 0x7f97b41069c0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97ab7fe640 1 -- 172.21.15.78:0/508118173 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ac0046c0 con 0x7f97b41069c0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b419d580 con 0x7f97b41069c0 2024-09-16T01:19:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97ab7fe640 1 -- 172.21.15.78:0/508118173 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac046480 con 0x7f97b41069c0 2024-09-16T01:19:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.324+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41a4e70 con 0x7f97b41069c0 2024-09-16T01:19:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.326+0000 7f97ab7fe640 1 -- 172.21.15.78:0/508118173 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f97ac0305a0 con 0x7f97b41069c0 2024-09-16T01:19:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.326+0000 7f97bf5f4740 1 Processor -- start 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 -- start start 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a55fa10 0x55a27a55fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a27a4c42f0 con 0x55a27a560330 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a27a46c270 con 0x55a27a56f0c0 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a27a4c02b0 con 0x55a27a55fa10 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97ba51d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97ba51d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41376/0 (socket says 172.21.15.78:41376) 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97ba51d640 1 -- 172.21.15.78:0/2933337855 learned_addr learned my addr 172.21.15.78:0/2933337855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97ab7fe640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 0x7f97a00a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a55fa10 0x55a27a55fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.327+0000 7f97bad1e640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 0x7f97a00a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 -- 172.21.15.78:0/2933337855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a55fa10 msgr2=0x55a27a55fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a55fa10 0x55a27a55fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 -- 172.21.15.78:0/2933337855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 msgr2=0x55a27a5714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 -- 172.21.15.78:0/2933337855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27a46b340 con 0x55a27a560330 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ab7fe640 1 -- 172.21.15.78:0/508118173 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f97ac02e030 con 0x7f97b41069c0 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f97a400bba0 tx=0x7f97a400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8def496879be0ed7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97bad1e640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 0x7f97a00a5920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0bd800 tx=0x7f97ac0a8050 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97a8ff9640 1 -- 172.21.15.78:0/2933337855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4017620 con 0x55a27a560330 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97a8ff9640 1 -- 172.21.15.78:0/2933337855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a4017ca0 con 0x55a27a560330 2024-09-16T01:19:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.328+0000 7f97a8ff9640 1 -- 172.21.15.78:0/2933337855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a400fd80 con 0x55a27a560330 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2933337855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 msgr2=0x55a27a56eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f97a400bba0 tx=0x7f97a400d710 comp rx=0 tx=0).stop 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2933337855 shutdown_connections 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a560330 0x55a27a56eb80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2933337855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a55fa10 0x55a27a55fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2933337855 >> 172.21.15.78:0/2933337855 conn(0x55a27a55c060 msgr2=0x55a27a55c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2933337855 shutdown_connections 2024-09-16T01:19:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2933337855 wait complete. 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 -- start start 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.329+0000 7f97bf5f4740 1 Processor -- start 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a55fa10 0x55a27a56bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a56c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a27a571c90 con 0x55a27a56f0c0 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a27a44d290 con 0x55a27a55fa10 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a27a571e10 con 0x55a27a560330 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97bf5f4740 10 client.0 _ll_register_callbacks cb 0x55a27a47e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41386/0 (socket says 172.21.15.78:41386) 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 -- 172.21.15.78:0/1647864479 learned_addr learned my addr 172.21.15.78:0/1647864479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97ba51d640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a56c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a55fa10 0x55a27a56bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 -- 172.21.15.78:0/1647864479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 msgr2=0x55a27a56c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a56c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 -- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a55fa10 msgr2=0x55a27a56bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a55fa10 0x55a27a56bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 -- 172.21.15.78:0/1647864479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27a1d7210 con 0x55a27a56f0c0 2024-09-16T01:19:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f97b0012950 tx=0x7f97b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f322f8372e5416 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.330+0000 7f979dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b00108d0 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f97b0010a70 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a4017ca0 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97b0010c70 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 10 client.16383 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a27a2a3620 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a4017ca0 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f97b001e3c0 con 0x55a27a56f0c0 2024-09-16T01:19:43.342 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 10 client.16383 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 20 client.16383 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a57f020 0x55a27a581410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 4 client.16383 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.331+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a27a4d1a20 con 0x55a27a57f020 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.332+0000 7f97ba51d640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a57f020 0x55a27a581410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.332+0000 7f97ba51d640 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a57f020 0x55a27a581410 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf13463a38855fa9 server_cookie=417da3d9b92af325 in_seq=0 out_seq=0 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.332+0000 7f979dffb640 10 client.16383 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f979dffb640 1 -- 172.21.15.78:0/1647864479 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a27a4d1a20 con 0x55a27a57f020 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f979dffb640 10 client.16383 handle_command_reply: tid=0 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 client.16383 shutdown 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 msgr2=0x55a27a56c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.343 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f97b0012950 tx=0x7f97b00046c0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 shutdown_connections 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a57f020 0x55a27a581410 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a56f0c0 0x55a27a56c9f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a56c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f979dffb640 0 client.16383 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/1647864479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a55fa10 0x55a27a56bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 >> 172.21.15.78:0/1647864479 conn(0x55a27a55c060 msgr2=0x55a27a561d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 shutdown_connections 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 1 -- 172.21.15.78:0/1647864479 wait complete. 2024-09-16T01:19:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.333+0000 7f97bf5f4740 20 client.16383 trim_cache size 0 max 16384 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 Processor -- start 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 -- start start 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a587220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41af610 con 0x55a27a55fa10 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41acee0 con 0x55a27a56f0c0 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41aad50 con 0x55a27a560330 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41396/0 (socket says 172.21.15.78:41396) 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.348+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2790243350 learned_addr learned my addr 172.21.15.78:0/2790243350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a587220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2790243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 msgr2=0x55a27a587220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a587220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2790243350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 msgr2=0x55a27a5691b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5691b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2790243350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27a560f20 con 0x55a27a55fa10 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a587220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f97b000ba30 tx=0x7f97b0013fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a64687da27f1b8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f979ffff640 1 -- 172.21.15.78:0/2790243350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0034070 con 0x55a27a55fa10 2024-09-16T01:19:43.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f979ffff640 1 -- 172.21.15.78:0/2790243350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0010e10 con 0x55a27a55fa10 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.349+0000 7f979ffff640 1 -- 172.21.15.78:0/2790243350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0005040 con 0x55a27a55fa10 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2790243350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 msgr2=0x55a27a586ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f97b000ba30 tx=0x7f97b0013fa0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2790243350 shutdown_connections 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a5691b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a587220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2790243350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a586ce0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2790243350 >> 172.21.15.78:0/2790243350 conn(0x55a27a55c060 msgr2=0x55a27a55b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2790243350 shutdown_connections 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2790243350 wait complete. 2024-09-16T01:19:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 -- start start 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.350+0000 7f97bf5f4740 1 Processor -- start 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a573650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a573bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41ab3a0 con 0x55a27a55fa10 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41ac810 con 0x55a27a56f0c0 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f97b41aba70 con 0x55a27a560330 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 10 client.0 _ll_register_callbacks cb 0x7f97b000b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41408/0 (socket says 172.21.15.78:41408) 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2078953638 learned_addr learned my addr 172.21.15.78:0/2078953638 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a573650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2078953638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 msgr2=0x55a27a573650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a573650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 msgr2=0x55a27a573bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a573bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 -- 172.21.15.78:0/2078953638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27a5e8f80 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97b9d1c640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005890 tx=0x7f97b000b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ff13413697d8c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f979dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0034070 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f97b0005040 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.351+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b0010e10 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97b0042050 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 10 client.16389 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a27a5ea200 con 0x55a27a55fa10 2024-09-16T01:19:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0010e10 con 0x55a27a55fa10 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f97b0003c90 con 0x55a27a55fa10 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 10 client.16389 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 20 client.16389 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a5ea860 0x55a27a5ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 4 client.16389 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a27a4cbd70 con 0x55a27a5ea860 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.352+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a5ea860 0x55a27a5ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.353+0000 7f979dffb640 10 client.16389 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.353+0000 7f97ba51d640 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a5ea860 0x55a27a5ecc50 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27b1d56c799ee2e1 server_cookie=50fc1401e71c283f in_seq=0 out_seq=0 2024-09-16T01:19:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f979dffb640 1 -- 172.21.15.78:0/2078953638 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a27a4cbd70 con 0x55a27a5ea860 2024-09-16T01:19:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f979dffb640 10 client.16389 handle_command_reply: tid=0 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 client.16389 shutdown 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 msgr2=0x55a27a573110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005890 tx=0x7f97b000b400 comp rx=0 tx=0).stop 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 shutdown_connections 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a27a5ea860 0x55a27a5ecc50 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27a56f0c0 0x55a27a573bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f979dffb640 0 client.16389 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a27a560330 0x55a27a573650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 --2- 172.21.15.78:0/2078953638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a27a55fa10 0x55a27a573110 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.360+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 >> 172.21.15.78:0/2078953638 conn(0x55a27a55c060 msgr2=0x55a27a56bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 shutdown_connections 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bf5f4740 1 -- 172.21.15.78:0/2078953638 wait complete. 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bf5f4740 20 client.16389 trim_cache size 0 max 16384 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:19:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 msgr2=0x7f97a00a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 0x7f97a00a5920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0bd800 tx=0x7f97ac0a8050 comp rx=0 tx=0).stop 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 msgr2=0x7f97b419cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005cc0 tx=0x7f97ac003970 comp rx=0 tx=0).stop 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 shutdown_connections 2024-09-16T01:19:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97a00a3460 0x7f97a00a5920 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b41069c0 0x7f97b419cbc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97b4101350 0x7f97b4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 --2- 172.21.15.78:0/508118173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97b41009b0 0x7f97b4198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 >> 172.21.15.78:0/508118173 conn(0x7f97b4077990 msgr2=0x7f97b40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 shutdown_connections 2024-09-16T01:19:43.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:43.361+0000 7f97bc781640 1 -- 172.21.15.78:0/508118173 wait complete. 2024-09-16T01:19:43.413 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- 172.21.15.78:0/1717618014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 msgr2=0x7fb4d0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 --2- 172.21.15.78:0/1717618014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0100ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009930 tx=0x7fb4b802a420 comp rx=0 tx=0).stop 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- 172.21.15.78:0/1717618014 shutdown_connections 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 --2- 172.21.15.78:0/1717618014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4d0101420 0x7fb4d010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 --2- 172.21.15.78:0/1717618014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0100ee0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 --2- 172.21.15.78:0/1717618014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0106a80 0x7fb4d0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- 172.21.15.78:0/1717618014 >> 172.21.15.78:0/1717618014 conn(0x7fb4d00fc7a0 msgr2=0x7fb4d00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- 172.21.15.78:0/1717618014 shutdown_connections 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- 172.21.15.78:0/1717618014 wait complete. 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 Processor -- start 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.965+0000 7fb4d6789640 1 -- start start 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 0x7fb4d01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4d0106a80 0x7fb4d01a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0110ea0 con 0x7fb4d0100a80 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0110d20 con 0x7fb4d0106a80 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0111020 con 0x7fb4d0101420 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41426/0 (socket says 172.21.15.78:41426) 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 -- 172.21.15.78:0/736224311 learned_addr learned my addr 172.21.15.78:0/736224311 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cf7fe640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 0x7fb4d01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 -- 172.21.15.78:0/736224311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 msgr2=0x7fb4d01997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 0x7fb4d01997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 -- 172.21.15.78:0/736224311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4d0106a80 msgr2=0x7fb4d01a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4d0106a80 0x7fb4d01a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 -- 172.21.15.78:0/736224311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d01a4850 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cf7fe640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 0x7fb4d01997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cffff640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000c920 tx=0x7fb4c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0007c70 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c0007e10 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d01a4a20 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.966+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0018890 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.967+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d01a4ee0 con 0x7fb4d0100a80 2024-09-16T01:19:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.968+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d0100ee0 con 0x7fb4d0100a80 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.969+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb4c0018a50 con 0x7fb4d0100a80 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.970+0000 7fb4cd7fa640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 0x7fb4bc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.970+0000 7fb4cf7fe640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 0x7fb4bc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.970+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fb4c00d05b0 con 0x7fb4d0100a80 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.970+0000 7fb4cf7fe640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 0x7fb4bc0a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8000c00 tx=0x7fb4b8005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:43.973+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb4c0014030 con 0x7fb4d0100a80 2024-09-16T01:19:44.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.173+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4d01a5210 con 0x7fb4d0100a80 2024-09-16T01:19:44.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.174+0000 7fb4cd7fa640 1 -- 172.21.15.78:0/736224311 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fb4c009cc30 con 0x7fb4d0100a80 2024-09-16T01:19:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:19:44.176 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:19:44.176 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:19:44.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 msgr2=0x7fb4bc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:44.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 0x7fb4bc0a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8000c00 tx=0x7fb4b8005a20 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 msgr2=0x7fb4d0199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000c920 tx=0x7fb4c000cde0 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 shutdown_connections 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb4bc0a33b0 0x7fb4bc0a5870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4d0106a80 0x7fb4d01a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0101420 0x7fb4d01997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 --2- 172.21.15.78:0/736224311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4d0100a80 0x7fb4d0199290 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 >> 172.21.15.78:0/736224311 conn(0x7fb4d00fc7a0 msgr2=0x7fb4d00fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 shutdown_connections 2024-09-16T01:19:44.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:44.176+0000 7fb4d6789640 1 -- 172.21.15.78:0/736224311 wait complete. 2024-09-16T01:19:44.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:44 smithi078 ceph-mon[30937]: pgmap v277: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2024-09-16T01:19:44.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:44 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/736224311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:44.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:44 smithi193 ceph-mon[35701]: pgmap v277: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2024-09-16T01:19:44.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:44 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/736224311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:44 smithi148 ceph-mon[35030]: pgmap v277: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2024-09-16T01:19:44.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:44 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/736224311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:44.924 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id d69f434f --release --await' 2024-09-16T01:19:44.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id d69f434f --release --await 2024-09-16T01:19:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- 172.21.15.78:0/3278275800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad881027a0 msgr2=0x7fad881067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3278275800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad881027a0 0x7fad881067f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fad70009900 tx=0x7fad7002a410 comp rx=0 tx=0).stop 2024-09-16T01:19:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- 172.21.15.78:0/3278275800 shutdown_connections 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3278275800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 0x7fad8810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3278275800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad881027a0 0x7fad881067f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3278275800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad88101df0 0x7fad881021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- 172.21.15.78:0/3278275800 >> 172.21.15.78:0/3278275800 conn(0x7fad880fd8f0 msgr2=0x7fad880ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- 172.21.15.78:0/3278275800 shutdown_connections 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- 172.21.15.78:0/3278275800 wait complete. 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 Processor -- start 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.688+0000 7fad8cd92640 1 -- start start 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad881027a0 0x7fad88198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 0x7fad8819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fad88111fc0 con 0x7fad88101df0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fad88111e40 con 0x7fad88106f20 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fad88112140 con 0x7fad881027a0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50982/0 (socket says 172.21.15.78:50982) 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 -- 172.21.15.78:0/3274280780 learned_addr learned my addr 172.21.15.78:0/3274280780 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 0x7fad8819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad7ffff640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad881027a0 0x7fad88198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 -- 172.21.15.78:0/3274280780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad881027a0 msgr2=0x7fad88198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad881027a0 0x7fad88198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 -- 172.21.15.78:0/3274280780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 msgr2=0x7fad8819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 0x7fad8819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 -- 172.21.15.78:0/3274280780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8819d4a0 con 0x7fad88101df0 2024-09-16T01:19:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8659c640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fad7800d770 tx=0x7fad7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad7f7fe640 1 -- 172.21.15.78:0/3274280780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78017070 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad7f7fe640 1 -- 172.21.15.78:0/3274280780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad78005500 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8819d790 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad7f7fe640 1 -- 172.21.15.78:0/3274280780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78004e00 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.689+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad881a5070 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad7f7fe640 1 -- 172.21.15.78:0/3274280780 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fad78004fa0 con 0x7fad88101df0 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 Processor -- start 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 -- start start 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee35440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 0x559e0ee3fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559e0ed9a620 con 0x559e0ee40240 2024-09-16T01:19:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e0ed41e50 con 0x559e0ee35060 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.692+0000 7fad8f16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559e0ed965e0 con 0x559e0ee35980 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50994/0 (socket says 172.21.15.78:50994) 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 -- 172.21.15.78:0/2285632135 learned_addr learned my addr 172.21.15.78:0/2285632135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7f7fe640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 0x7fad740a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad86d9d640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee35440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 0x7fad740a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7f7fe640 1 -- 172.21.15.78:0/3274280780 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fad780cfca0 con 0x7fad88101df0 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad8659c640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 0x559e0ee3fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 -- 172.21.15.78:0/2285632135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 msgr2=0x559e0ee3fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 0x559e0ee3fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 -- 172.21.15.78:0/2285632135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 msgr2=0x559e0ee35440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee35440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 -- 172.21.15.78:0/2285632135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e0ed32d40 con 0x559e0ee40240 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad86d9d640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee35440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad8659c640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 0x559e0ee3fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7ffff640 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fad70006010 tx=0x7fad70005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926580c3c23de65e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7cff9640 1 -- 172.21.15.78:0/2285632135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7003f070 con 0x559e0ee40240 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7cff9640 1 -- 172.21.15.78:0/2285632135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad70004030 con 0x559e0ee40240 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad7cff9640 1 -- 172.21.15.78:0/2285632135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70035070 con 0x559e0ee40240 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.693+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 0x7fad740a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fad800070b0 tx=0x7fad8000c040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 -- 172.21.15.78:0/2285632135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 msgr2=0x559e0ee42630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fad70006010 tx=0x7fad70005e80 comp rx=0 tx=0).stop 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 -- 172.21.15.78:0/2285632135 shutdown_connections 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee42630 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35980 0x559e0ee3fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2285632135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee35440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 -- 172.21.15.78:0/2285632135 >> 172.21.15.78:0/2285632135 conn(0x559e0ee315f0 msgr2=0x559e0ee319d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 -- 172.21.15.78:0/2285632135 shutdown_connections 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.694+0000 7fad8f16b740 1 -- 172.21.15.78:0/2285632135 wait complete. 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 -- start start 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 Processor -- start 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 0x559e0ee3d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559e0ee42e10 con 0x559e0ee40240 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e0ed22f10 con 0x559e0ee35980 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559e0ee42f90 con 0x559e0ee35060 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8f16b740 10 client.0 _ll_register_callbacks cb 0x559e0ed46db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:45.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 0x559e0ee3d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51008/0 (socket says 172.21.15.78:51008) 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad8659c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 0x559e0ee3d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58804/0 (socket says 172.21.15.78:58804) 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad86d9d640 1 -- 172.21.15.78:0/3117967705 learned_addr learned my addr 172.21.15.78:0/3117967705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.695+0000 7fad7ffff640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 -- 172.21.15.78:0/3117967705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 msgr2=0x559e0ee3d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 0x559e0ee3d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 -- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 msgr2=0x559e0ee3d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 -- 172.21.15.78:0/3117967705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e0eaa9580 con 0x559e0ee40240 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad7ffff640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad86d9d640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fad80017bd0 tx=0x7fad80004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fd1e5e8bb6723b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80027060 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fad80022400 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad70004030 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fad8002e050 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad8f16b740 10 client.16407 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559e0eb73960 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad70004030 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fad80021670 con 0x559e0ee40240 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad8f16b740 10 client.16407 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.696+0000 7fad8f16b740 10 client.16407 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8f16b740 20 client.16407 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:45.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee4fd20 0x559e0ee52170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8f16b740 4 client.16407 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e0eda7b90 con 0x559e0ee4fd20 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8659c640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee4fd20 0x559e0ee52170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad8659c640 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee4fd20 0x559e0ee52170 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c777d4357a3e150f server_cookie=8f2cec235fd4cb2b in_seq=0 out_seq=0 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.697+0000 7fad6dffb640 10 client.16407 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad6dffb640 1 -- 172.21.15.78:0/3117967705 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e0eda7b90 con 0x559e0ee4fd20 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad6dffb640 10 client.16407 handle_command_reply: tid=0 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 client.16407 shutdown 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 msgr2=0x559e0ee3daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fad80017bd0 tx=0x7fad80004b50 comp rx=0 tx=0).stop 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 shutdown_connections 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee4fd20 0x559e0ee52170 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee40240 0x559e0ee3daf0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 --2- 172.21.15.78:0/3117967705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee35060 0x559e0ee3d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad6dffb640 0 client.16407 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.698+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 >> 172.21.15.78:0/3117967705 conn(0x559e0ee315f0 msgr2=0x559e0ee372d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.699+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 shutdown_connections 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.699+0000 7fad8f16b740 1 -- 172.21.15.78:0/3117967705 wait complete. 2024-09-16T01:19:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.699+0000 7fad8f16b740 20 client.16407 trim_cache size 0 max 16384 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 Processor -- start 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 -- start start 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee57a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fad881b1a10 con 0x559e0ee35060 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fad881af2e0 con 0x559e0ee35980 2024-09-16T01:19:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad8f16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ad150 con 0x559e0ee315f0 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad7ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51014/0 (socket says 172.21.15.78:51014) 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.715+0000 7fad7ffff640 1 -- 172.21.15.78:0/516573239 learned_addr learned my addr 172.21.15.78:0/516573239 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad8659c640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee57a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad86d9d640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 -- 172.21.15.78:0/516573239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 msgr2=0x559e0ee57a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee57a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 -- 172.21.15.78:0/516573239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 msgr2=0x559e0ee3a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 -- 172.21.15.78:0/516573239 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e0ee4e8c0 con 0x559e0ee35060 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad86d9d640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad7ffff640 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fad70006010 tx=0x7fad70032e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b18a00fef1b560 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad6ffff640 1 -- 172.21.15.78:0/516573239 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7003f070 con 0x559e0ee35060 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad6ffff640 1 -- 172.21.15.78:0/516573239 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad7004c070 con 0x559e0ee35060 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.716+0000 7fad6ffff640 1 -- 172.21.15.78:0/516573239 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70035040 con 0x559e0ee35060 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- 172.21.15.78:0/516573239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 msgr2=0x559e0ee57f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fad70006010 tx=0x7fad70032e30 comp rx=0 tx=0).stop 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- 172.21.15.78:0/516573239 shutdown_connections 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35980 0x559e0ee3a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35060 0x559e0ee57f80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- 172.21.15.78:0/516573239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee57a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- 172.21.15.78:0/516573239 >> 172.21.15.78:0/516573239 conn(0x559e0ee38a60 msgr2=0x559e0ee3b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- 172.21.15.78:0/516573239 shutdown_connections 2024-09-16T01:19:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- 172.21.15.78:0/516573239 wait complete. 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- start start 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 Processor -- start 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee44df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ad7a0 con 0x559e0ee35980 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fad881aec10 con 0x559e0ee35060 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.717+0000 7fad8f16b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fad881ade70 con 0x559e0ee315f0 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad8f16b740 10 client.0 _ll_register_callbacks cb 0x7fad80017790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51018/0 (socket says 172.21.15.78:51018) 2024-09-16T01:19:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 -- 172.21.15.78:0/2535568861 learned_addr learned my addr 172.21.15.78:0/2535568861 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad8659c640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad7ffff640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee44df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 -- 172.21.15.78:0/2535568861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 msgr2=0x559e0ee448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 -- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 msgr2=0x559e0ee44df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee44df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 -- 172.21.15.78:0/2535568861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e0ee31420 con 0x559e0ee35980 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad86d9d640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fad8000af60 tx=0x7fad80021f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556c1439ff98e4fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80036020 con 0x559e0ee35980 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fad800215a0 con 0x559e0ee35980 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad7004c070 con 0x559e0ee35980 2024-09-16T01:19:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fad80027020 con 0x559e0ee35980 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad8f16b740 10 client.16413 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.718+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559e0eebb2a0 con 0x559e0ee35980 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad7004c070 con 0x559e0ee35980 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fad80032b40 con 0x559e0ee35980 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 10 client.16413 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 10 client.16413 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 20 client.16413 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee40240 0x559e0eebdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 4 client.16413 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d69f434f", "release": true, "await": true}] 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d69f434f", "release": true, "await": true}) -- 0x559e0ee2c950 con 0x559e0ee40240 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8659c640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee40240 0x559e0eebdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad8659c640 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee40240 0x559e0eebdab0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b6383ad77268875 server_cookie=a7ce85b8b022ca26 in_seq=0 out_seq=0 2024-09-16T01:19:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.719+0000 7fad6dffb640 10 client.16413 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.779+0000 7fad6dffb640 1 -- 172.21.15.78:0/2535568861 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x559e0ee2c950 con 0x559e0ee40240 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.779+0000 7fad6dffb640 10 client.16413 handle_command_reply: tid=0 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.779+0000 7fad8f16b740 1 client.16413 shutdown 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.779+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 msgr2=0x559e0ee47eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.779+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fad8000af60 tx=0x7fad80021f60 comp rx=0 tx=0).stop 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 shutdown_connections 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559e0ee40240 0x559e0eebdab0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e0ee35980 0x559e0ee47eb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e0ee35060 0x559e0ee44df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad6dffb640 0 client.16413 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 --2- 172.21.15.78:0/2535568861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559e0ee315f0 0x559e0ee448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 >> 172.21.15.78:0/2535568861 conn(0x559e0ee38a60 msgr2=0x559e0ee36ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 shutdown_connections 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 1 -- 172.21.15.78:0/2535568861 wait complete. 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.780+0000 7fad8f16b740 20 client.16413 trim_cache size 0 max 16384 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 32, 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "d69f434f": { 2024-09-16T01:19:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 32, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 142.3, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:19:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:45.785 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:19:45.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 msgr2=0x7fad740a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 0x7fad740a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fad800070b0 tx=0x7fad8000c040 comp rx=0 tx=0).stop 2024-09-16T01:19:45.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 msgr2=0x7fad881984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:45.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fad7800d770 tx=0x7fad7800dc30 comp rx=0 tx=0).stop 2024-09-16T01:19:45.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 shutdown_connections 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fad740a33b0 0x7fad740a5870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad88106f20 0x7fad8819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad881027a0 0x7fad88198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 --2- 172.21.15.78:0/3274280780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad88101df0 0x7fad881984f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 >> 172.21.15.78:0/3274280780 conn(0x7fad880fd8f0 msgr2=0x7fad880fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 shutdown_connections 2024-09-16T01:19:45.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:45.781+0000 7fad8cd92640 1 -- 172.21.15.78:0/3274280780 wait complete. 2024-09-16T01:19:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 510.0 s, 12.4 tps, lat 17790.388 ms stddev 8237.861 2024-09-16T01:19:46.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:46 smithi078 ceph-mon[30937]: pgmap v278: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-09-16T01:19:46.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:46 smithi193 ceph-mon[35701]: pgmap v278: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-09-16T01:19:46.869 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'd69f434f', total seconds elapsed: 26.1 2024-09-16T01:19:46.870 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-16T01:19:46.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:19:46.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:46 smithi148 ceph-mon[35030]: pgmap v278: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2024-09-16T01:19:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1948257536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff458105500 msgr2=0x7ff45810d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1948257536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff458105500 0x7ff45810d080 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff448009900 tx=0x7ff44802b650 comp rx=0 tx=0).stop 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1948257536 shutdown_connections 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1948257536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff458105500 0x7ff45810d080 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1948257536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff458100c70 0x7ff458104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1948257536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4581002c0 0x7ff4581006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.967+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1948257536 >> 172.21.15.78:0/1948257536 conn(0x7ff458077010 msgr2=0x7ff4580fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.968+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1948257536 shutdown_connections 2024-09-16T01:19:47.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.972+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1948257536 wait complete. 2024-09-16T01:19:47.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.973+0000 7ff45eeb0640 1 Processor -- start 2024-09-16T01:19:47.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.973+0000 7ff45eeb0640 1 -- start start 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.973+0000 7ff45eeb0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4581002c0 0x7ff4581961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.974+0000 7ff45eeb0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff458100c70 0x7ff4581966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.974+0000 7ff45eeb0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.974+0000 7ff45eeb0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff45810fdc0 con 0x7ff4581002c0 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.974+0000 7ff45eeb0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff45810fc40 con 0x7ff458100c70 2024-09-16T01:19:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.974+0000 7ff45eeb0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff45810ff40 con 0x7ff458105500 2024-09-16T01:19:47.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:47.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58840/0 (socket says 172.21.15.78:58840) 2024-09-16T01:19:47.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 -- 172.21.15.78:0/1352285576 learned_addr learned my addr 172.21.15.78:0/1352285576 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:47.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45deae640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4581002c0 0x7ff4581961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 -- 172.21.15.78:0/1352285576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff458100c70 msgr2=0x7ff4581966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff458100c70 0x7ff4581966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 -- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4581002c0 msgr2=0x7ff4581961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4581002c0 0x7ff4581961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.975+0000 7ff45e6af640 1 -- 172.21.15.78:0/1352285576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45819b150 con 0x7ff458105500 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff45e6af640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff448031940 tx=0x7ff448031970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44803f020 con 0x7ff458105500 2024-09-16T01:19:47.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45819b3e0 con 0x7ff458105500 2024-09-16T01:19:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45819b850 con 0x7ff458105500 2024-09-16T01:19:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4480035f0 con 0x7ff458105500 2024-09-16T01:19:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.976+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448036070 con 0x7ff458105500 2024-09-16T01:19:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.978+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4581017a0 con 0x7ff458105500 2024-09-16T01:19:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.981+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7ff448042020 con 0x7ff458105500 2024-09-16T01:19:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.982+0000 7ff44effd640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 0x7ff4440a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.982+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7ff4480f1c80 con 0x7ff458105500 2024-09-16T01:19:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.982+0000 7ff45deae640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 0x7ff4440a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.983+0000 7ff45deae640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 0x7ff4440a9dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff450009f40 tx=0x7ff4500094d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:47.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:47.987+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4480f8080 con 0x7ff458105500 2024-09-16T01:19:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.325+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff458197500 con 0x7ff458105500 2024-09-16T01:19:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.328+0000 7ff44effd640 1 -- 172.21.15.78:0/1352285576 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7ff448049020 con 0x7ff458105500 2024-09-16T01:19:48.330 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:19:48.330 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:19:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:19:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 msgr2=0x7ff4440a9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 0x7ff4440a9dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff450009f40 tx=0x7ff4500094d0 comp rx=0 tx=0).stop 2024-09-16T01:19:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 msgr2=0x7ff45819aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff448031940 tx=0x7ff448031970 comp rx=0 tx=0).stop 2024-09-16T01:19:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 shutdown_connections 2024-09-16T01:19:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7ff4440a7900 0x7ff4440a9dc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff458105500 0x7ff45819aa70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff458100c70 0x7ff4581966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:48.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 --2- 172.21.15.78:0/1352285576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4581002c0 0x7ff4581961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.337+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 >> 172.21.15.78:0/1352285576 conn(0x7ff458077010 msgr2=0x7ff458104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.339+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 shutdown_connections 2024-09-16T01:19:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:48.340+0000 7ff45eeb0640 1 -- 172.21.15.78:0/1352285576 wait complete. 2024-09-16T01:19:48.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:48 smithi078 ceph-mon[30937]: pgmap v279: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 32 KiB/s wr, 10 op/s 2024-09-16T01:19:48.761 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:48 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1352285576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:48.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:48 smithi193 ceph-mon[35701]: pgmap v279: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 32 KiB/s wr, 10 op/s 2024-09-16T01:19:48.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:48 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1352285576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:48.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:48 smithi148 ceph-mon[35030]: pgmap v279: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 32 KiB/s wr, 10 op/s 2024-09-16T01:19:48.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:48 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1352285576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:19:49.120 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-16T01:19:49.120 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-16T01:19:49.543 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:19:50.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:50 smithi078 ceph-mon[30937]: pgmap v280: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 499 KiB/s wr, 70 op/s 2024-09-16T01:19:50.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:50 smithi193 ceph-mon[35701]: pgmap v280: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 499 KiB/s wr, 70 op/s 2024-09-16T01:19:50.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:50 smithi148 ceph-mon[35030]: pgmap v280: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 499 KiB/s wr, 70 op/s 2024-09-16T01:19:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 515.0 s, 518.6 tps, lat 591.892 ms stddev 2965.221 2024-09-16T01:19:52.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:52 smithi078 ceph-mon[30937]: pgmap v281: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-09-16T01:19:52.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:52 smithi193 ceph-mon[35701]: pgmap v281: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-09-16T01:19:52.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:52 smithi148 ceph-mon[35030]: pgmap v281: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 213 op/s 2024-09-16T01:19:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 -- 172.21.15.78:0/3903338143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc11b140 msgr2=0x7f85cc11d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/3903338143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc11b140 0x7f85cc11d530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009be0 tx=0x7f85c402a6c0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 -- 172.21.15.78:0/3903338143 shutdown_connections 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/3903338143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc11b140 0x7f85cc11d530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/3903338143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc109830 0x7f85cc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/3903338143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85cc109450 0x7f85cc110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 -- 172.21.15.78:0/3903338143 >> 172.21.15.78:0/3903338143 conn(0x7f85cc06e140 msgr2=0x7f85cc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 -- 172.21.15.78:0/3903338143 shutdown_connections 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.330+0000 7f85d1cdc640 1 -- 172.21.15.78:0/3903338143 wait complete. 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.331+0000 7f85d1cdc640 1 Processor -- start 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.331+0000 7f85d1cdc640 1 -- start start 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85cc109450 0x7f85cc1a71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc109830 0x7f85cc1a7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc120d50 con 0x7f85cc109450 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc120bd0 con 0x7f85cc11b140 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.332+0000 7f85d1cdc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc120ed0 con 0x7f85cc109830 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85caffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc109830 0x7f85cc1a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43698/0 (socket says 172.21.15.78:43698) 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 -- 172.21.15.78:0/2384595515 learned_addr learned my addr 172.21.15.78:0/2384595515 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 -- 172.21.15.78:0/2384595515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc109830 msgr2=0x7f85cc1a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc109830 0x7f85cc1a7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 -- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85cc109450 msgr2=0x7f85cc1a71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85cc109450 0x7f85cc1a71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.333+0000 7f85cbfff640 1 -- 172.21.15.78:0/2384595515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc1ac180 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85cbfff640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f85c4034040 tx=0x7f85c4030c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85c8ff9640 1 -- 172.21.15.78:0/2384595515 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4041020 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85c8ff9640 1 -- 172.21.15.78:0/2384595515 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c4037040 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc1ac410 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc1b3d50 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.334+0000 7f85c8ff9640 1 -- 172.21.15.78:0/2384595515 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4036070 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85c8ff9640 1 -- 172.21.15.78:0/2384595515 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f85c4044020 con 0x7f85cc11b140 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85d40b5740 1 Processor -- start 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85d40b5740 1 -- start start 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e32004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.337+0000 7f85caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43714/0 (socket says 172.21.15.78:43714) 2024-09-16T01:19:54.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e3200a30 0x5604e3202e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 -- 172.21.15.78:0/1711573798 learned_addr learned my addr 172.21.15.78:0/1711573798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5604e315a160 con 0x5604e31f6170 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5604e3101720 con 0x5604e31f5850 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5604e3156120 con 0x5604e3200a30 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e32004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85c8ff9640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 0x7f85b40a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85cbfff640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 0x7f85b40a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e3200a30 0x5604e3202e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 -- 172.21.15.78:0/1711573798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e3200a30 msgr2=0x5604e3202e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e3200a30 0x5604e3202e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 -- 172.21.15.78:0/1711573798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 msgr2=0x5604e32004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e32004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 -- 172.21.15.78:0/1711573798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e2ecbba0 con 0x5604e31f5850 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85c8ff9640 1 -- 172.21.15.78:0/2384595515 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f85c40f0c60 con 0x7f85cc11b140 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85caffd640 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f85c0004650 tx=0x7f85c0009980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=725d01d5a6d485c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.338+0000 7f85cbfff640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 0x7f85b40a59c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85c40c5230 tx=0x7f85c40b8e70 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.339+0000 7f85b1ffb640 1 -- 172.21.15.78:0/1711573798 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c001a030 con 0x5604e31f5850 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.339+0000 7f85b1ffb640 1 -- 172.21.15.78:0/1711573798 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c0014170 con 0x5604e31f5850 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.339+0000 7f85b1ffb640 1 -- 172.21.15.78:0/1711573798 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0011820 con 0x5604e31f5850 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.339+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 msgr2=0x5604e31f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.339+0000 7f85d40b5740 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f85c0004650 tx=0x7f85c0009980 comp rx=0 tx=0).stop 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 shutdown_connections 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e3200a30 0x5604e3202e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e32004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 --2- 172.21.15.78:0/1711573798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e31f5c30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 >> 172.21.15.78:0/1711573798 conn(0x5604e31f1de0 msgr2=0x5604e31f21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 shutdown_connections 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.340+0000 7f85d40b5740 1 -- 172.21.15.78:0/1711573798 wait complete. 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.341+0000 7f85d40b5740 1 -- start start 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 Processor -- start 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 0x5604e31fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31fe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5604e3203600 con 0x5604e3200a30 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5604e30e2900 con 0x5604e31f6170 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5604e3203780 con 0x5604e31f5850 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85caffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 0x5604e31fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85caffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58874/0 (socket says 172.21.15.78:58874) 2024-09-16T01:19:54.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85caffd640 1 -- 172.21.15.78:0/3263943671 learned_addr learned my addr 172.21.15.78:0/3263943671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85caffd640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31fe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.342+0000 7f85d40b5740 10 client.0 _ll_register_callbacks cb 0x5604e30e0780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 -- 172.21.15.78:0/3263943671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 msgr2=0x5604e31fd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 -- 172.21.15.78:0/3263943671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 msgr2=0x5604e31fe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31fe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 -- 172.21.15.78:0/3263943671 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e3122c50 con 0x5604e31f6170 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 0x5604e31fde60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0114b0 tx=0x7f85bc011970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d35eb09c8dcd0fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc004030 con 0x5604e31f6170 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f85bc0041d0 con 0x5604e31f6170 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.343+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c0014170 con 0x5604e31f6170 2024-09-16T01:19:54.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f85bc024050 con 0x5604e31f6170 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 10 client.25808 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604e320f300 con 0x5604e31f6170 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c0014170 con 0x5604e31f6170 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f85bc0038d0 con 0x5604e31f6170 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 10 client.25808 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 10 client.25808 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 20 client.25808 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e3210860 0x5604e3212c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 4 client.25808 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.344+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604e3167890 con 0x5604e3210860 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.345+0000 7f85caffd640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e3210860 0x5604e3212c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.345+0000 7f85caffd640 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e3210860 0x5604e3212c80 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f65e79038f44321 server_cookie=c87532ed6af39f43 in_seq=0 out_seq=0 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.345+0000 7f85aaffd640 10 client.25808 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:54.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.346+0000 7f85aaffd640 1 -- 172.21.15.78:0/3263943671 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604e3167890 con 0x5604e3210860 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.346+0000 7f85aaffd640 10 client.25808 handle_command_reply: tid=0 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.346+0000 7f85d40b5740 1 client.25808 shutdown 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.346+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 msgr2=0x5604e31fde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.346+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 0x5604e31fde60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0114b0 tx=0x7f85bc011970 comp rx=0 tx=0).stop 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 shutdown_connections 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e3210860 0x5604e3212c80 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f6170 0x5604e31fde60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 --2- 172.21.15.78:0/3263943671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 >> 172.21.15.78:0/3263943671 conn(0x5604e31f1de0 msgr2=0x5604e31f7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.348+0000 7f85aaffd640 0 client.25808 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.349+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 shutdown_connections 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.349+0000 7f85d40b5740 1 -- 172.21.15.78:0/3263943671 wait complete. 2024-09-16T01:19:54.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.349+0000 7f85d40b5740 20 client.25808 trim_cache size 0 max 16384 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.372+0000 7f85d40b5740 1 Processor -- start 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.379+0000 7f85d40b5740 1 -- start start 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e3218530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f6170 0x5604e3218a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31faba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1c0750 con 0x5604e3200a30 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1be020 con 0x5604e31f5850 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.380+0000 7f85d40b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1bbe90 con 0x5604e31f6170 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31faba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31faba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51096/0 (socket says 172.21.15.78:51096) 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 -- 172.21.15.78:0/182816712 learned_addr learned my addr 172.21.15.78:0/182816712 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e3218530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 -- 172.21.15.78:0/182816712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f6170 msgr2=0x5604e3218a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f6170 0x5604e3218a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 -- 172.21.15.78:0/182816712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 msgr2=0x5604e31faba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31faba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.387+0000 7f85caffd640 1 -- 172.21.15.78:0/182816712 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e31f5fa0 con 0x5604e31f5850 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.389+0000 7f85caffd640 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e3218530 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85c0010f50 tx=0x7f85c0011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c726588fb06a64bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.393+0000 7f85b17fa640 1 -- 172.21.15.78:0/182816712 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0033030 con 0x5604e31f5850 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.393+0000 7f85b17fa640 1 -- 172.21.15.78:0/182816712 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c001b040 con 0x5604e31f5850 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.393+0000 7f85b17fa640 1 -- 172.21.15.78:0/182816712 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c001a040 con 0x5604e31f5850 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 -- 172.21.15.78:0/182816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 msgr2=0x5604e3218530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e3218530 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85c0010f50 tx=0x7f85c0011440 comp rx=0 tx=0).stop 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 -- 172.21.15.78:0/182816712 shutdown_connections 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e3200a30 0x5604e31faba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f6170 0x5604e3218a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 --2- 172.21.15.78:0/182816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e31f5850 0x5604e3218530 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.395+0000 7f85d40b5740 1 -- 172.21.15.78:0/182816712 >> 172.21.15.78:0/182816712 conn(0x5604e31f1de0 msgr2=0x5604e31ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.398+0000 7f85d40b5740 1 -- 172.21.15.78:0/182816712 shutdown_connections 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.402+0000 7f85d40b5740 1 -- 172.21.15.78:0/182816712 wait complete. 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.402+0000 7f85d40b5740 1 -- start start 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.402+0000 7f85d40b5740 1 Processor -- start 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e31fad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 0x5604e31fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1bc4e0 con 0x5604e31f6170 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1bd950 con 0x5604e3200a30 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1bcbb0 con 0x5604e31f5850 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.403+0000 7f85d40b5740 10 client.0 _ll_register_callbacks cb 0x7f85bc009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:54.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.404+0000 7f85caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 0x5604e31fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.404+0000 7f85caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 0x5604e31fb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:43746/0 (socket says 172.21.15.78:43746) 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.404+0000 7f85caffd640 1 -- 172.21.15.78:0/4168652267 learned_addr learned my addr 172.21.15.78:0/4168652267 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.404+0000 7f85caffd640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.405+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e31fad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.406+0000 7f85cb7fe640 1 -- 172.21.15.78:0/4168652267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 msgr2=0x5604e31fa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.406+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.406+0000 7f85cb7fe640 1 -- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 msgr2=0x5604e31fb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.406+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 0x5604e31fb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.406+0000 7f85cb7fe640 1 -- 172.21.15.78:0/4168652267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e327a500 con 0x5604e31f6170 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.407+0000 7f85cb7fe640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e31fad70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x5604e31f6620 tx=0x7f85bc009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff01121542cf0d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.407+0000 7f85aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.409+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc02c070 con 0x5604e31f6170 2024-09-16T01:19:54.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.409+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7f85bc0037f0 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.409+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c001b040 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.417+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f85bc036050 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.418+0000 7f85d40b5740 10 client.16434 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.418+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604e329d6d0 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c001b040 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f85bc00fa70 con 0x5604e31f6170 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 10 client.16434 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 10 client.16434 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 20 client.16434 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e329de30 0x5604e32a0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 4 client.16434 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.422+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x5604e3161be0 con 0x5604e329de30 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.423+0000 7f85caffd640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e329de30 0x5604e32a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:54.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.426+0000 7f85caffd640 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e329de30 0x5604e32a0250 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688a12335f3b50dd server_cookie=9c642a00f6751340 in_seq=0 out_seq=0 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.426+0000 7f85aaffd640 10 client.16434 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85aaffd640 1 -- 172.21.15.78:0/4168652267 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5604e3161be0 con 0x5604e329de30 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85aaffd640 10 client.16434 handle_command_reply: tid=0 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 client.16434 shutdown 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 msgr2=0x5604e31fad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e31fad70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x5604e31f6620 tx=0x7f85bc009320 comp rx=0 tx=0).stop 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:19:54.196 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 34, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "9889a58a": { 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 34, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 142.3, 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:19:54.197 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:19:54.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 shutdown_connections 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5604e329de30 0x5604e32a0250 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5604e3200a30 0x5604e31fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85aaffd640 0 client.16434 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604e31f6170 0x5604e31fad70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 --2- 172.21.15.78:0/4168652267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604e31f5850 0x5604e31fa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 >> 172.21.15.78:0/4168652267 conn(0x5604e31f1de0 msgr2=0x5604e31fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 shutdown_connections 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.795+0000 7f85d40b5740 1 -- 172.21.15.78:0/4168652267 wait complete. 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d40b5740 20 client.16434 trim_cache size 0 max 16384 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 msgr2=0x7f85b40a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 0x7f85b40a59c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85c40c5230 tx=0x7f85c40b8e70 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 msgr2=0x7f85cc1abaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f85c4034040 tx=0x7f85c4030c40 comp rx=0 tx=0).stop 2024-09-16T01:19:54.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 shutdown_connections 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f85b40a3500 0x7f85b40a59c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f85cc11b140 0x7f85cc1abaa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85cc109830 0x7f85cc1a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 --2- 172.21.15.78:0/2384595515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85cc109450 0x7f85cc1a71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 >> 172.21.15.78:0/2384595515 conn(0x7f85cc06e140 msgr2=0x7f85cc11c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 shutdown_connections 2024-09-16T01:19:54.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:19:53.796+0000 7f85d1cdc640 1 -- 172.21.15.78:0/2384595515 wait complete. 2024-09-16T01:19:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:54 smithi078 ceph-mon[30937]: pgmap v282: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-16T01:19:54.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:54 smithi193 ceph-mon[35701]: pgmap v282: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-16T01:19:54.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:54 smithi148 ceph-mon[35030]: pgmap v282: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-09-16T01:19:55.419 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9889a58a', quiesce took 8.5 seconds. Will release after: 13.850000000000001 2024-09-16T01:19:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 520.0 s, 241.5 tps, lat 217.001 ms stddev 278.585 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- 172.21.15.78:0/3739051867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744101350 msgr2=0x7fe74410d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 --2- 172.21.15.78:0/3739051867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744101350 0x7fe74410d880 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009cd0 tx=0x7fe73c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- 172.21.15.78:0/3739051867 shutdown_connections 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 --2- 172.21.15.78:0/3739051867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744101350 0x7fe74410d880 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 --2- 172.21.15.78:0/3739051867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7441009b0 0x7fe744100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 --2- 172.21.15.78:0/3739051867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441069c0 0x7fe744106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- 172.21.15.78:0/3739051867 >> 172.21.15.78:0/3739051867 conn(0x7fe744077990 msgr2=0x7fe7440fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- 172.21.15.78:0/3739051867 shutdown_connections 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- 172.21.15.78:0/3739051867 wait complete. 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 Processor -- start 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.452+0000 7fe74abe4640 1 -- start start 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 0x7fe744199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101350 0x7fe7441995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe744110e50 con 0x7fe7441069c0 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe744110cd0 con 0x7fe7441009b0 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe744110fd0 con 0x7fe744101350 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35314/0 (socket says 172.21.15.78:35314) 2024-09-16T01:19:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 -- 172.21.15.78:0/814407992 learned_addr learned my addr 172.21.15.78:0/814407992 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe743fff640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101350 0x7fe7441995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe748980640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 0x7fe744199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 -- 172.21.15.78:0/814407992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101350 msgr2=0x7fe7441995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101350 0x7fe7441995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 -- 172.21.15.78:0/814407992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 msgr2=0x7fe744199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 0x7fe744199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 -- 172.21.15.78:0/814407992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7441a4640 con 0x7fe7441069c0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe748980640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 0x7fe744199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe749181640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009ca0 tx=0x7fe73c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe741ffb640 1 -- 172.21.15.78:0/814407992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c004240 con 0x7fe7441069c0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe741ffb640 1 -- 172.21.15.78:0/814407992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73c0305a0 con 0x7fe7441069c0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7441a4810 con 0x7fe7441069c0 2024-09-16T01:19:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.453+0000 7fe741ffb640 1 -- 172.21.15.78:0/814407992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c03a660 con 0x7fe7441069c0 2024-09-16T01:19:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.454+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441a4d00 con 0x7fe7441069c0 2024-09-16T01:19:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe741ffb640 1 -- 172.21.15.78:0/814407992 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fe73c030750 con 0x7fe7441069c0 2024-09-16T01:19:56.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 Processor -- start 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 -- start start 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc99410 0x5562afca3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afca3cb0 0x5562afca60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5562afbfd2f0 con 0x5562afc98af0 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5562afba5270 con 0x5562afc99410 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.456+0000 7fe74da5b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5562afbf92b0 con 0x5562afca3cb0 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35318/0 (socket says 172.21.15.78:35318) 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 -- 172.21.15.78:0/1839520554 learned_addr learned my addr 172.21.15.78:0/1839520554 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe748980640 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc99410 0x5562afca3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe741ffb640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 0x7fe7340a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afca3cb0 0x5562afca60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 -- 172.21.15.78:0/1839520554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afca3cb0 msgr2=0x5562afca60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afca3cb0 0x5562afca60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 -- 172.21.15.78:0/1839520554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc99410 msgr2=0x5562afca3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc99410 0x5562afca3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe749181640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 0x7fe7340a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 -- 172.21.15.78:0/1839520554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562afba4340 con 0x5562afc98af0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe741ffb640 1 -- 172.21.15.78:0/814407992 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fe73c02e030 con 0x7fe7441069c0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe743fff640 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe73000cfa0 tx=0x7fe73000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5baa36ba1936a609 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe72effd640 1 -- 172.21.15.78:0/1839520554 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730018030 con 0x5562afc98af0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe72effd640 1 -- 172.21.15.78:0/1839520554 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe730005500 con 0x5562afc98af0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.457+0000 7fe72effd640 1 -- 172.21.15.78:0/1839520554 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7300108a0 con 0x5562afc98af0 2024-09-16T01:19:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- 172.21.15.78:0/1839520554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 msgr2=0x5562afc98ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe73000cfa0 tx=0x7fe73000c620 comp rx=0 tx=0).stop 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe749181640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 0x7fe7340a5920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0bd000 tx=0x7fe73c08f080 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- 172.21.15.78:0/1839520554 shutdown_connections 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afca3cb0 0x5562afca60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc99410 0x5562afca3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 --2- 172.21.15.78:0/1839520554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc98ed0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- 172.21.15.78:0/1839520554 >> 172.21.15.78:0/1839520554 conn(0x5562afc95060 msgr2=0x5562afc95440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- 172.21.15.78:0/1839520554 shutdown_connections 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- 172.21.15.78:0/1839520554 wait complete. 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 -- start start 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.458+0000 7fe74da5b740 1 Processor -- start 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc98af0 0x5562afca0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5562afca6880 con 0x5562afca3cb0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5562afb86290 con 0x5562afc98af0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5562afca6a00 con 0x5562afc99410 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe74da5b740 10 client.0 _ll_register_callbacks cb 0x5562afbb7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35330/0 (socket says 172.21.15.78:35330) 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 -- 172.21.15.78:0/239507592 learned_addr learned my addr 172.21.15.78:0/239507592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc98af0 0x5562afca0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe748980640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 -- 172.21.15.78:0/239507592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 msgr2=0x5562afca1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 -- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc98af0 msgr2=0x5562afca0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc98af0 0x5562afca0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 -- 172.21.15.78:0/239507592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562af910210 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe748980640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe743fff640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe730010a20 tx=0x7fe730004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a7d1fa2cf69d46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe723fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730021070 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.459+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fe730010a80 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe730005500 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe730033050 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5562af9dc620 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe730005500 con 0x5562afca3cb0 2024-09-16T01:19:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe73001d6a0 con 0x5562afca3cb0 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 20 client.16446 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afcb3b60 0x5562afcb5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 4 client.16446 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562afc0aa20 con 0x5562afcb3b60 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.460+0000 7fe748980640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afcb3b60 0x5562afcb5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.461+0000 7fe723fff640 10 client.16446 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.461+0000 7fe748980640 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afcb3b60 0x5562afcb5f50 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6340851134bfd7f7 server_cookie=cc85adb20689adbb in_seq=0 out_seq=0 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.461+0000 7fe723fff640 1 -- 172.21.15.78:0/239507592 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5562afc0aa20 con 0x5562afcb3b60 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe723fff640 10 client.16446 handle_command_reply: tid=0 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 client.16446 shutdown 2024-09-16T01:19:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 msgr2=0x5562afca15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe730010a20 tx=0x7fe730004f30 comp rx=0 tx=0).stop 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 shutdown_connections 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afcb3b60 0x5562afcb5f50 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afca3cb0 0x5562afca15a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe723fff640 0 client.16446 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 --2- 172.21.15.78:0/239507592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afc98af0 0x5562afca0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 >> 172.21.15.78:0/239507592 conn(0x5562afc95060 msgr2=0x5562afc9adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 shutdown_connections 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 1 -- 172.21.15.78:0/239507592 wait complete. 2024-09-16T01:19:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.462+0000 7fe74da5b740 20 client.16446 trim_cache size 0 max 16384 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 Processor -- start 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 -- start start 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afcbb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441b1830 con 0x5562afc98af0 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441af100 con 0x5562afca3cb0 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe74da5b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441acf70 con 0x5562afc99410 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.476+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35344/0 (socket says 172.21.15.78:35344) 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 -- 172.21.15.78:0/3527976451 learned_addr learned my addr 172.21.15.78:0/3527976451 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 -- 172.21.15.78:0/3527976451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 msgr2=0x5562afca13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 -- 172.21.15.78:0/3527976451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 msgr2=0x5562afcbb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afcbb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 -- 172.21.15.78:0/3527976451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562afc94e90 con 0x5562afc98af0 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe743fff640 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe730004ff0 tx=0x7fe73000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c8dab193f96a66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe72e7fc640 1 -- 172.21.15.78:0/3527976451 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73003c070 con 0x5562afc98af0 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe72e7fc640 1 -- 172.21.15.78:0/3527976451 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe730030030 con 0x5562afc98af0 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe72e7fc640 1 -- 172.21.15.78:0/3527976451 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe730021040 con 0x5562afc98af0 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe74da5b740 1 -- 172.21.15.78:0/3527976451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 msgr2=0x5562afc9b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.477+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe730004ff0 tx=0x7fe73000c910 comp rx=0 tx=0).stop 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- 172.21.15.78:0/3527976451 shutdown_connections 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afcbb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc99410 0x5562afca13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3527976451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc98af0 0x5562afc9b560 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- 172.21.15.78:0/3527976451 >> 172.21.15.78:0/3527976451 conn(0x5562afc95060 msgr2=0x5562afca2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- 172.21.15.78:0/3527976451 shutdown_connections 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- 172.21.15.78:0/3527976451 wait complete. 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- start start 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 Processor -- start 2024-09-16T01:19:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc98af0 0x5562afd16860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afca72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441ad5c0 con 0x5562afc99410 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441aea30 con 0x5562afca3cb0 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441adc90 con 0x5562afc98af0 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe74da5b740 10 client.0 _ll_register_callbacks cb 0x7fe738005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe748980640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe748980640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35356/0 (socket says 172.21.15.78:35356) 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.478+0000 7fe748980640 1 -- 172.21.15.78:0/3002808429 learned_addr learned my addr 172.21.15.78:0/3002808429 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 -- 172.21.15.78:0/3002808429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc98af0 msgr2=0x5562afd16860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc98af0 0x5562afd16860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 -- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 msgr2=0x5562afca72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afca72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 -- 172.21.15.78:0/3002808429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562afc96110 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe748980640 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe738005e20 tx=0x7fe7380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91eafffeeeefddee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73801e020 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9525+0+0 (secure 0 0 0) 0x7fe738005520 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe730030030 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe738029050 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 10 client.16452 fetch_fsmap learned FSMap version 31 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5562afd1df00 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe730030030 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe73801a830 con 0x5562afc99410 2024-09-16T01:19:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 10 client.16452 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 20 client.16452 populate_metadata read hostname 'smithi078' 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afd1e6b0 0x5562afd20ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 4 client.16452 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.479+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562afc04d70 con 0x5562afd1e6b0 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.480+0000 7fe743fff640 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afd1e6b0 0x5562afd20ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.480+0000 7fe743fff640 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afd1e6b0 0x5562afd20ad0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5154392c1b9b64bd server_cookie=5661f1b397001797 in_seq=0 out_seq=0 2024-09-16T01:19:56.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.480+0000 7fe723fff640 10 client.16452 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe723fff640 1 -- 172.21.15.78:0/3002808429 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562afc04d70 con 0x5562afd1e6b0 2024-09-16T01:19:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe723fff640 10 client.16452 handle_command_reply: tid=0 2024-09-16T01:19:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 client.16452 shutdown 2024-09-16T01:19:56.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 msgr2=0x5562afd16da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe738005e20 tx=0x7fe7380118d0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 shutdown_connections 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5562afd1e6b0 0x5562afd20ad0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5562afca3cb0 0x5562afca72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5562afc99410 0x5562afd16da0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe723fff640 0 client.16452 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 --2- 172.21.15.78:0/3002808429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5562afc98af0 0x5562afd16860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 >> 172.21.15.78:0/3002808429 conn(0x5562afc95060 msgr2=0x5562afca2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 shutdown_connections 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 1 -- 172.21.15.78:0/3002808429 wait complete. 2024-09-16T01:19:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.485+0000 7fe74da5b740 20 client.16452 trim_cache size 0 max 16384 2024-09-16T01:19:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:19:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:19:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:19:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 msgr2=0x7fe7340a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 0x7fe7340a5920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0bd000 tx=0x7fe73c08f080 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 msgr2=0x7fe7441a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009ca0 tx=0x7fe73c003ba0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 shutdown_connections 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe7340a3460 0x7fe7340a5920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7441069c0 0x7fe7441a4100 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe744101350 0x7fe7441995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 --2- 172.21.15.78:0/814407992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7441009b0 0x7fe744199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 >> 172.21.15.78:0/814407992 conn(0x7fe744077990 msgr2=0x7fe744103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:19:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 shutdown_connections 2024-09-16T01:19:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:19:56.486+0000 7fe74abe4640 1 -- 172.21.15.78:0/814407992 wait complete. 2024-09-16T01:19:56.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:56 smithi078 ceph-mon[30937]: pgmap v283: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T01:19:56.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:56 smithi193 ceph-mon[35701]: pgmap v283: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T01:19:56.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:56 smithi148 ceph-mon[35030]: pgmap v283: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 MiB/s wr, 387 op/s 2024-09-16T01:19:58.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:58 smithi078 ceph-mon[30937]: pgmap v284: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-09-16T01:19:58.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:58 smithi193 ceph-mon[35701]: pgmap v284: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-09-16T01:19:58.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:58 smithi148 ceph-mon[35030]: pgmap v284: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 3.5 MiB/s wr, 407 op/s 2024-09-16T01:19:59.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:19:59 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:19:59.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:19:59 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:19:59.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:19:59 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:20:00.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:00 smithi078 ceph-mon[30937]: pgmap v285: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:20:00.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:00 smithi078 ceph-mon[30937]: overall HEALTH_OK 2024-09-16T01:20:00.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:00 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:20:00.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:00 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:20:00.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:00 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:20:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:00 smithi193 ceph-mon[35701]: pgmap v285: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:20:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:00 smithi193 ceph-mon[35701]: overall HEALTH_OK 2024-09-16T01:20:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:00 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:20:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:00 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:20:00.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:00 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:20:00.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:00 smithi148 ceph-mon[35030]: pgmap v285: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 406 op/s 2024-09-16T01:20:00.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:00 smithi148 ceph-mon[35030]: overall HEALTH_OK 2024-09-16T01:20:00.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:00 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:20:00.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:00 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:20:00.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:00 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:20:00.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 525.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:20:02.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:02 smithi078 ceph-mon[30937]: pgmap v286: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-09-16T01:20:02.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:02 smithi193 ceph-mon[35701]: pgmap v286: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-09-16T01:20:02.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:02 smithi148 ceph-mon[35030]: pgmap v286: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-09-16T01:20:04.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:04 smithi078 ceph-mon[30937]: pgmap v287: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:04.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:04 smithi193 ceph-mon[35701]: pgmap v287: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:04.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:04 smithi148 ceph-mon[35030]: pgmap v287: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 530.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:20:06.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:06 smithi078 ceph-mon[30937]: pgmap v288: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:06.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:06 smithi193 ceph-mon[35701]: pgmap v288: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:06.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:06 smithi148 ceph-mon[35030]: pgmap v288: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-09-16T01:20:08.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:08 smithi193 ceph-mon[35701]: pgmap v289: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 234 KiB/s wr, 45 op/s 2024-09-16T01:20:08.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:08 smithi148 ceph-mon[35030]: pgmap v289: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 234 KiB/s wr, 45 op/s 2024-09-16T01:20:09.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:08 smithi078 ceph-mon[30937]: pgmap v289: 129 pgs: 129 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 234 KiB/s wr, 45 op/s 2024-09-16T01:20:09.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:20:09.518 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:20:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 -- 172.21.15.78:0/160568818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 msgr2=0x7f4384100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 --2- 172.21.15.78:0/160568818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f4384100e10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4374009930 tx=0x7f437402a420 comp rx=0 tx=0).stop 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 -- 172.21.15.78:0/160568818 shutdown_connections 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 --2- 172.21.15.78:0/160568818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 0x7f438410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 --2- 172.21.15.78:0/160568818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f4384100e10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 --2- 172.21.15.78:0/160568818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f4384106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 -- 172.21.15.78:0/160568818 >> 172.21.15.78:0/160568818 conn(0x7f4384077990 msgr2=0x7f43840fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 -- 172.21.15.78:0/160568818 shutdown_connections 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.576+0000 7f438b96f640 1 -- 172.21.15.78:0/160568818 wait complete. 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 Processor -- start 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 -- start start 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 0x7f43841996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f43841a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4384110e00 con 0x7f43841009b0 2024-09-16T01:20:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4384110c80 con 0x7f43841069c0 2024-09-16T01:20:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438b96f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4384110f80 con 0x7f4384101350 2024-09-16T01:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41472/0 (socket says 172.21.15.78:41472) 2024-09-16T01:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 -- 172.21.15.78:0/4135481675 learned_addr learned my addr 172.21.15.78:0/4135481675 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f4388f0a640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 0x7f43841996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 -- 172.21.15.78:0/4135481675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 msgr2=0x7f43841996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 0x7f43841996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f4389f0c640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f43841a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 -- 172.21.15.78:0/4135481675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 msgr2=0x7f43841a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f43841a4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 -- 172.21.15.78:0/4135481675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43841a4760 con 0x7f43841009b0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f4389f0c640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f43841a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.577+0000 7f438970b640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f438000d950 tx=0x7f438000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.578+0000 7f437a7fc640 1 -- 172.21.15.78:0/4135481675 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380017030 con 0x7f43841009b0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.578+0000 7f437a7fc640 1 -- 172.21.15.78:0/4135481675 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380004020 con 0x7f43841009b0 2024-09-16T01:20:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.578+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43841a4930 con 0x7f43841009b0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.578+0000 7f437a7fc640 1 -- 172.21.15.78:0/4135481675 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43800047a0 con 0x7f43841009b0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.578+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43841a4e50 con 0x7f43841009b0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.580+0000 7f437a7fc640 1 -- 172.21.15.78:0/4135481675 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f43800041d0 con 0x7f43841009b0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.580+0000 7f438e7e2740 1 Processor -- start 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.580+0000 7f438e7e2740 1 -- start start 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244968770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b24496b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b2448c22f0 con 0x55b24495daf0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b24486a270 con 0x55b244968cb0 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438e7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b2448be2b0 con 0x55b24495e410 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f437a7fc640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 0x7f43700a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41482/0 (socket says 172.21.15.78:41482) 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 -- 172.21.15.78:0/3362575452 learned_addr learned my addr 172.21.15.78:0/3362575452 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 0x7f43700a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4388f0a640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b24496b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f437a7fc640 1 -- 172.21.15.78:0/4135481675 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f43800cfb70 con 0x7f43841009b0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438970b640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244968770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 -- 172.21.15.78:0/3362575452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 msgr2=0x55b244968770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244968770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 -- 172.21.15.78:0/3362575452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 msgr2=0x55b24496b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b24496b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 -- 172.21.15.78:0/3362575452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b244869340 con 0x55b24495daf0 2024-09-16T01:20:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f438970b640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244968770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f4389f0c640 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f437c0149b0 tx=0x7f437c014e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c3a091c5ba39e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f436f7fe640 1 -- 172.21.15.78:0/3362575452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c021070 con 0x55b24495daf0 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f436f7fe640 1 -- 172.21.15.78:0/3362575452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437c00f8b0 con 0x55b24495daf0 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.581+0000 7f436f7fe640 1 -- 172.21.15.78:0/3362575452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c01b5b0 con 0x55b24495daf0 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 -- 172.21.15.78:0/3362575452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 msgr2=0x55b24495ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f4389f0c640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 0x7f43700a5920 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f437c01e000 tx=0x7f437c01b000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f437c0149b0 tx=0x7f437c014e70 comp rx=0 tx=0).stop 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 -- 172.21.15.78:0/3362575452 shutdown_connections 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b24496b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244968770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 --2- 172.21.15.78:0/3362575452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b24495ded0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 -- 172.21.15.78:0/3362575452 >> 172.21.15.78:0/3362575452 conn(0x55b24495a060 msgr2=0x55b24495a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 -- 172.21.15.78:0/3362575452 shutdown_connections 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.582+0000 7f438e7e2740 1 -- 172.21.15.78:0/3362575452 wait complete. 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 -- start start 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 Processor -- start 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244966090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b2449665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b24496b880 con 0x55b24495daf0 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b24484b290 con 0x55b244968cb0 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b24496ba00 con 0x55b24495e410 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438e7e2740 10 client.0 _ll_register_callbacks cb 0x55b24487c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41494/0 (socket says 172.21.15.78:41494) 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 -- 172.21.15.78:0/2980432312 learned_addr learned my addr 172.21.15.78:0/2980432312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f4389f0c640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b2449665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 -- 172.21.15.78:0/2980432312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 msgr2=0x55b244966090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f4388f0a640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244966090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244966090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 -- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 msgr2=0x55b2449665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b2449665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f438970b640 1 -- 172.21.15.78:0/2980432312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2445d5210 con 0x55b24495daf0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.583+0000 7f4389f0c640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b2449665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438970b640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f438009b9a0 tx=0x7f438009b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55852a8e5d469703 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438009fdc0 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f4380098080 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f437c00f8b0 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f43800a4680 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 10 client.16464 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b2446a1620 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437c00f8b0 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f438009ae60 con 0x55b24495daf0 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 10 client.16464 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 20 client.16464 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b24497af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 4 client.16464 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.584+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2448cfa20 con 0x55b244978b30 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.585+0000 7f4388f0a640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b24497af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.585+0000 7f4388f0a640 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b24497af60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a23c7c85d93b5ec server_cookie=c9489263d1e69a37 in_seq=0 out_seq=0 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.585+0000 7f436cff9640 10 client.16464 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f436cff9640 1 -- 172.21.15.78:0/2980432312 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b2448cfa20 con 0x55b244978b30 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f436cff9640 10 client.16464 handle_command_reply: tid=0 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 client.16464 shutdown 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 msgr2=0x55b244965b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f438009b9a0 tx=0x7f438009b680 comp rx=0 tx=0).stop 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 shutdown_connections 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b24497af60 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f436cff9640 0 client.16464 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:09.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b2449665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b244966090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 --2- 172.21.15.78:0/2980432312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244965b50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 >> 172.21.15.78:0/2980432312 conn(0x55b24495a060 msgr2=0x55b2449591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 shutdown_connections 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 1 -- 172.21.15.78:0/2980432312 wait complete. 2024-09-16T01:20:09.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.586+0000 7f438e7e2740 20 client.16464 trim_cache size 0 max 16384 2024-09-16T01:20:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.600+0000 7f438e7e2740 1 Processor -- start 2024-09-16T01:20:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.600+0000 7f438e7e2740 1 -- start start 2024-09-16T01:20:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.600+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b2449808c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b244980e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438e7e2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f43841b1a80 con 0x55b24495daf0 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438e7e2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43841af350 con 0x55b244968cb0 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438e7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43841ad1c0 con 0x55b24495e410 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41502/0 (socket says 172.21.15.78:41502) 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 -- 172.21.15.78:0/1741122374 learned_addr learned my addr 172.21.15.78:0/1741122374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 -- 172.21.15.78:0/1741122374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 msgr2=0x55b2449808c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b2449808c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 -- 172.21.15.78:0/1741122374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 msgr2=0x55b244980e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b244980e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 -- 172.21.15.78:0/1741122374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b244961700 con 0x55b24495daf0 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f438970b640 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4380097e70 tx=0x7f4380000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5665039b2a0cfcab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f436effd640 1 -- 172.21.15.78:0/1741122374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380014070 con 0x55b24495daf0 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f436effd640 1 -- 172.21.15.78:0/1741122374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380064020 con 0x55b24495daf0 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.601+0000 7f436effd640 1 -- 172.21.15.78:0/1741122374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43800a42a0 con 0x55b24495daf0 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- 172.21.15.78:0/1741122374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 msgr2=0x55b244966400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4380097e70 tx=0x7f4380000bc0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- 172.21.15.78:0/1741122374 shutdown_connections 2024-09-16T01:20:09.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b244968cb0 0x55b244980e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b24495e410 0x55b2449808c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1741122374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b244966400 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- 172.21.15.78:0/1741122374 >> 172.21.15.78:0/1741122374 conn(0x55b24495a060 msgr2=0x55b244967bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- 172.21.15.78:0/1741122374 shutdown_connections 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- 172.21.15.78:0/1741122374 wait complete. 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- start start 2024-09-16T01:20:09.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 Processor -- start 2024-09-16T01:20:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 0x55b24496c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 0x55b24496cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f43841ad810 con 0x55b24495daf0 2024-09-16T01:20:09.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43841aec80 con 0x55b24495e410 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f43841adee0 con 0x55b244968cb0 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.602+0000 7f438e7e2740 10 client.0 _ll_register_callbacks cb 0x7f4380004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41510/0 (socket says 172.21.15.78:41510) 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 -- 172.21.15.78:0/1881282649 learned_addr learned my addr 172.21.15.78:0/1881282649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f4388f0a640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 0x55b24496c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f4389f0c640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 0x55b24496cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 -- 172.21.15.78:0/1881282649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 msgr2=0x55b24496cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 0x55b24496cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 -- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 msgr2=0x55b24496c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 0x55b24496c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 -- 172.21.15.78:0/1881282649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24496d460 con 0x55b24495daf0 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f4389f0c640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 0x55b24496cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f4388f0a640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 0x55b24496c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:20:09.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438970b640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f43800983e0 tx=0x7f43800ba550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269aab3cce561fe2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f436cff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438007f060 con 0x55b24495daf0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7f4380098080 con 0x55b24495daf0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4380064020 con 0x55b24495daf0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4380014030 con 0x55b24495daf0 2024-09-16T01:20:09.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438e7e2740 10 client.16470 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b2449e3890 con 0x55b24495daf0 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.603+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380064020 con 0x55b24495daf0 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f438009aa20 con 0x55b24495daf0 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 10 client.16470 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:09.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 20 client.16470 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b2449e5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 4 client.16470 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2448c9d70 con 0x55b244978b30 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f4388f0a640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b2449e5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f436cff9640 10 client.16470 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:09.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.604+0000 7f4388f0a640 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b2449e5e00 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36febdaa40b1ce7f server_cookie=6ef4237b80bc2e41 in_seq=0 out_seq=0 2024-09-16T01:20:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f436cff9640 1 -- 172.21.15.78:0/1881282649 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2448c9d70 con 0x55b244978b30 2024-09-16T01:20:09.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f436cff9640 10 client.16470 handle_command_reply: tid=0 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 client.16470 shutdown 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 msgr2=0x55b2449713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f43800983e0 tx=0x7f43800ba550 comp rx=0 tx=0).stop 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 shutdown_connections 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b244978b30 0x55b2449e5e00 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b244968cb0 0x55b24496cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f436cff9640 0 client.16470 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b24495e410 0x55b24496c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 --2- 172.21.15.78:0/1881282649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24495daf0 0x55b2449713d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 >> 172.21.15.78:0/1881282649 conn(0x55b24495a060 msgr2=0x55b2449589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 shutdown_connections 2024-09-16T01:20:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.612+0000 7f438e7e2740 1 -- 172.21.15.78:0/1881282649 wait complete. 2024-09-16T01:20:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438e7e2740 20 client.16470 trim_cache size 0 max 16384 2024-09-16T01:20:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:20:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:20:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:20:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:20:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 msgr2=0x7f43700a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 0x7f43700a5920 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f437c01e000 tx=0x7f437c01b000 comp rx=0 tx=0).stop 2024-09-16T01:20:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 msgr2=0x7f43841991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f438000d950 tx=0x7f438000de10 comp rx=0 tx=0).stop 2024-09-16T01:20:09.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 shutdown_connections 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f43700a3460 0x7f43700a5920 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43841069c0 0x7f43841a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4384101350 0x7f43841996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 --2- 172.21.15.78:0/4135481675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43841009b0 0x7f43841991a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 >> 172.21.15.78:0/4135481675 conn(0x7f4384077990 msgr2=0x7f43841037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 shutdown_connections 2024-09-16T01:20:09.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:09.613+0000 7f438b96f640 1 -- 172.21.15.78:0/4135481675 wait complete. 2024-09-16T01:20:09.641 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:09 smithi078 ceph-mon[30937]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T01:20:09.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:09 smithi193 ceph-mon[35701]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T01:20:09.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:09 smithi148 ceph-mon[35030]: osdmap e101: 12 total, 12 up, 12 in 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 -- 172.21.15.78:0/3183071045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102570 msgr2=0x7fc3241029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/3183071045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102570 0x7fc3241029b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc314009ca0 tx=0x7fc31402a950 comp rx=0 tx=0).stop 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 -- 172.21.15.78:0/3183071045 shutdown_connections 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/3183071045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102ef0 0x7fc32410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/3183071045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102570 0x7fc3241029b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/3183071045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc324101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 -- 172.21.15.78:0/3183071045 >> 172.21.15.78:0/3183071045 conn(0x7fc3240fc7d0 msgr2=0x7fc3240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 -- 172.21.15.78:0/3183071045 shutdown_connections 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.008+0000 7fc32aaf8640 1 -- 172.21.15.78:0/3183071045 wait complete. 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 Processor -- start 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 -- start start 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc3241993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 0x7fc324199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc324110fa0 con 0x7fc324102ef0 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc324110e20 con 0x7fc324100e20 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc32aaf8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc324111120 con 0x7fc324102570 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41538/0 (socket says 172.21.15.78:41538) 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 -- 172.21.15.78:0/938924857 learned_addr learned my addr 172.21.15.78:0/938924857 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc323fff640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 0x7fc324199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc328894640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc3241993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 -- 172.21.15.78:0/938924857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 msgr2=0x7fc324199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 0x7fc324199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 -- 172.21.15.78:0/938924857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 msgr2=0x7fc3241993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc3241993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.009+0000 7fc329095640 1 -- 172.21.15.78:0/938924857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3241a4990 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc323fff640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 0x7fc324199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc328894640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc3241993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc329095640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009fd0 tx=0x7fc30c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c011830 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc30c016070 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3241a4b60 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c00f2e0 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.010+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3241a5070 con 0x7fc324102ef0 2024-09-16T01:20:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.011+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc324101200 con 0x7fc324102ef0 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.012+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fc30c0119d0 con 0x7fc324102ef0 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.013+0000 7fc321ffb640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 0x7fc3100a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.013+0000 7fc328894640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 0x7fc3100a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.013+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fc30c0cf730 con 0x7fc324102ef0 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.013+0000 7fc328894640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 0x7fc3100a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005d00 tx=0x7fc31c009550 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.017+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc30c09bde0 con 0x7fc324102ef0 2024-09-16T01:20:10.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.219+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3241090c0 con 0x7fc324102ef0 2024-09-16T01:20:10.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.220+0000 7fc321ffb640 1 -- 172.21.15.78:0/938924857 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fc30c09b6e0 con 0x7fc324102ef0 2024-09-16T01:20:10.222 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:20:10.222 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:20:10.222 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.222+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 msgr2=0x7fc3100a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.222+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 0x7fc3100a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc31c005d00 tx=0x7fc31c009550 comp rx=0 tx=0).stop 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.222+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 msgr2=0x7fc3241a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.222+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009fd0 tx=0x7fc30c00b8e0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 shutdown_connections 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3100a33b0 0x7fc3100a5870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc324102ef0 0x7fc3241a4450 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc324102570 0x7fc324199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 --2- 172.21.15.78:0/938924857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc324100e20 0x7fc3241993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:10.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 >> 172.21.15.78:0/938924857 conn(0x7fc3240fc7d0 msgr2=0x7fc3240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:10.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 shutdown_connections 2024-09-16T01:20:10.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:10.223+0000 7fc32aaf8640 1 -- 172.21.15.78:0/938924857 wait complete. 2024-09-16T01:20:10.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:10 smithi078 ceph-mon[30937]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-09-16T01:20:10.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:10 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/938924857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:20:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:10 smithi193 ceph-mon[35701]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-09-16T01:20:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:10 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/938924857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:20:10.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:10 smithi148 ceph-mon[35030]: pgmap v291: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 58 KiB/s wr, 27 op/s 2024-09-16T01:20:10.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:10 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/938924857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:20:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 535.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:20:11.140 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 9889a58a --release --await' 2024-09-16T01:20:11.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 9889a58a --release --await 2024-09-16T01:20:11.400 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 -- 172.21.15.78:0/2544748938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 msgr2=0x7fa3e810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 --2- 172.21.15.78:0/2544748938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e810da40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009900 tx=0x7fa3e002a480 comp rx=0 tx=0).stop 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 -- 172.21.15.78:0/2544748938 shutdown_connections 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 --2- 172.21.15.78:0/2544748938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e810da40 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 --2- 172.21.15.78:0/2544748938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8102570 0x7fa3e81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 --2- 172.21.15.78:0/2544748938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8100e20 0x7fa3e8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 -- 172.21.15.78:0/2544748938 >> 172.21.15.78:0/2544748938 conn(0x7fa3e80fc7d0 msgr2=0x7fa3e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 -- 172.21.15.78:0/2544748938 shutdown_connections 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 -- 172.21.15.78:0/2544748938 wait complete. 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.954+0000 7fa3ee138640 1 Processor -- start 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 -- start start 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8100e20 0x7fa3e81991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 0x7fa3e8199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e8110f80 con 0x7fa3e8102ef0 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e8110e00 con 0x7fa3e8102570 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3ee138640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e8111100 con 0x7fa3e8100e20 2024-09-16T01:20:11.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41560/0 (socket says 172.21.15.78:41560) 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 -- 172.21.15.78:0/4013386678 learned_addr learned my addr 172.21.15.78:0/4013386678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8100e20 0x7fa3e81991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 0x7fa3e8199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 -- 172.21.15.78:0/4013386678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8100e20 msgr2=0x7fa3e81991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8100e20 0x7fa3e81991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 -- 172.21.15.78:0/4013386678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 msgr2=0x7fa3e8199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 0x7fa3e8199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 -- 172.21.15.78:0/4013386678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e81a47b0 con 0x7fa3e8102ef0 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 0x7fa3e8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e7fff640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0004770 tx=0x7fa3e0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.955+0000 7fa3e4ff9640 1 -- 172.21.15.78:0/4013386678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e003f070 con 0x7fa3e8102ef0 2024-09-16T01:20:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.956+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e81a4980 con 0x7fa3e8102ef0 2024-09-16T01:20:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.956+0000 7fa3e4ff9640 1 -- 172.21.15.78:0/4013386678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3e0031c30 con 0x7fa3e8102ef0 2024-09-16T01:20:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.956+0000 7fa3e4ff9640 1 -- 172.21.15.78:0/4013386678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0035030 con 0x7fa3e8102ef0 2024-09-16T01:20:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.956+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e81a4e10 con 0x7fa3e8102ef0 2024-09-16T01:20:11.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.958+0000 7fa3f0511740 1 Processor -- start 2024-09-16T01:20:11.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e4ff9640 1 -- 172.21.15.78:0/4013386678 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fa3e0042070 con 0x7fa3e8102ef0 2024-09-16T01:20:11.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 -- start start 2024-09-16T01:20:11.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f285110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f287a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55563f1db620 con 0x55563f275f20 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55563f182e50 con 0x55563f276840 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3f0511740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55563f1d75e0 con 0x55563f285650 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41570/0 (socket says 172.21.15.78:41570) 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/65258349 learned_addr learned my addr 172.21.15.78:0/65258349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e4ff9640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 0x7fa3d00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f285110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.959+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f287a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/65258349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 msgr2=0x55563f287a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f287a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/65258349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 msgr2=0x55563f285110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f285110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/65258349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55563f173d40 con 0x55563f275f20 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e7fff640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 0x7fa3d00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e4ff9640 1 -- 172.21.15.78:0/4013386678 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fa3e00f0a70 con 0x7fa3e8102ef0 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009fa0 tx=0x7fa3dc009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471a1fe8ebe2c3e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3cdffb640 1 -- 172.21.15.78:0/65258349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc005020 con 0x55563f275f20 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3cdffb640 1 -- 172.21.15.78:0/65258349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3dc0056a0 con 0x55563f275f20 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.960+0000 7fa3cdffb640 1 -- 172.21.15.78:0/65258349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc0136c0 con 0x55563f275f20 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- 172.21.15.78:0/65258349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 msgr2=0x55563f276300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3e7fff640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 0x7fa3d00a5870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00ede30 tx=0x7fa3e0047000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009fa0 tx=0x7fa3dc009590 comp rx=0 tx=0).stop 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- 172.21.15.78:0/65258349 shutdown_connections 2024-09-16T01:20:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f287a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f285110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 --2- 172.21.15.78:0/65258349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f276300 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- 172.21.15.78:0/65258349 >> 172.21.15.78:0/65258349 conn(0x55563f2725f0 msgr2=0x55563f2729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- 172.21.15.78:0/65258349 shutdown_connections 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- 172.21.15.78:0/65258349 wait complete. 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.961+0000 7fa3f0511740 1 -- start start 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 Processor -- start 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f282a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f282f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55563f2881c0 con 0x55563f275f20 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55563f163f10 con 0x55563f276840 2024-09-16T01:20:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55563f288340 con 0x55563f285650 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3f0511740 10 client.0 _ll_register_callbacks cb 0x55563f187db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41574/0 (socket says 172.21.15.78:41574) 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/571247193 learned_addr learned my addr 172.21.15.78:0/571247193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f282f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f282a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/571247193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 msgr2=0x55563f282f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f282f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 msgr2=0x55563f282a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f282a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/571247193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563eeea580 con 0x55563f275f20 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc018040 tx=0x7fa3dc004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a686b5e112056e0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.962+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc004220 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fa3dc0043c0 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3dc0056a0 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa3dc030050 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 10 client.16488 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55563efb4960 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3dc0056a0 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa3dc00c030 con 0x55563f275f20 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 10 client.16488 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 10 client.16488 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 20 client.16488 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f295120 0x55563f297510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 4 client.16488 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:20:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55563f1e8b90 con 0x55563f295120 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.963+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f295120 0x55563f297510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.964+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f295120 0x55563f297510 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185d75dc5e5441e2 server_cookie=87b30f47da9657e7 in_seq=0 out_seq=0 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.964+0000 7fa3c6ffd640 10 client.16488 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/571247193 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55563f1e8b90 con 0x55563f295120 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3c6ffd640 10 client.16488 handle_command_reply: tid=0 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 client.16488 shutdown 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 msgr2=0x55563f2824f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc018040 tx=0x7fa3dc004f30 comp rx=0 tx=0).stop 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 shutdown_connections 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f295120 0x55563f297510 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f282f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f282a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3c6ffd640 0 client.16488 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:11.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 --2- 172.21.15.78:0/571247193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f275f20 0x55563f2824f0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 >> 172.21.15.78:0/571247193 conn(0x55563f2725f0 msgr2=0x55563f271750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:11.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 shutdown_connections 2024-09-16T01:20:11.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 1 -- 172.21.15.78:0/571247193 wait complete. 2024-09-16T01:20:11.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.965+0000 7fa3f0511740 20 client.16488 trim_cache size 0 max 16384 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.981+0000 7fa3f0511740 1 Processor -- start 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 -- start start 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f290b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f29cb40 0x55563f29cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81af830 con 0x55563f285650 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81ad100 con 0x55563f276840 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3f0511740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81aaf70 con 0x55563f29cb40 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41590/0 (socket says 172.21.15.78:41590) 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 -- 172.21.15.78:0/1736375158 learned_addr learned my addr 172.21.15.78:0/1736375158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f290b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 -- 172.21.15.78:0/1736375158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f29cb40 msgr2=0x55563f29cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f29cb40 0x55563f29cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 -- 172.21.15.78:0/1736375158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 msgr2=0x55563f290b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f290b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 -- 172.21.15.78:0/1736375158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55563f270bb0 con 0x55563f285650 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f290b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80074b0 tx=0x7fa3d8010970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31b1a8d246e98aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3cd7fa640 1 -- 172.21.15.78:0/1736375158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8003c30 con 0x55563f285650 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3cd7fa640 1 -- 172.21.15.78:0/1736375158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3d8003dd0 con 0x55563f285650 2024-09-16T01:20:11.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.982+0000 7fa3cd7fa640 1 -- 172.21.15.78:0/1736375158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d801cab0 con 0x55563f285650 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- 172.21.15.78:0/1736375158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 msgr2=0x55563f2910d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa3d80074b0 tx=0x7fa3d8010970 comp rx=0 tx=0).stop 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- 172.21.15.78:0/1736375158 shutdown_connections 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f29cb40 0x55563f29cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f285650 0x55563f2910d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1736375158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f290b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- 172.21.15.78:0/1736375158 >> 172.21.15.78:0/1736375158 conn(0x55563f2725f0 msgr2=0x55563f290480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- 172.21.15.78:0/1736375158 shutdown_connections 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- 172.21.15.78:0/1736375158 wait complete. 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.983+0000 7fa3f0511740 1 -- start start 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 Processor -- start 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f28dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f289ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81ab5c0 con 0x55563f29cb40 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81aca30 con 0x55563f276840 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e81abc90 con 0x55563f285650 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3f0511740 10 client.0 _ll_register_callbacks cb 0x7fa3dc009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41604/0 (socket says 172.21.15.78:41604) 2024-09-16T01:20:11.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/1245621098 learned_addr learned my addr 172.21.15.78:0/1245621098 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f28dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f289ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/1245621098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 msgr2=0x55563f289ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f289ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 msgr2=0x55563f28dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f28dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.984+0000 7fa3e6ffd640 1 -- 172.21.15.78:0/1245621098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563f28a950 con 0x55563f29cb40 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3e6ffd640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x55563f283630 tx=0x7fa3dc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1286ef4667729289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc015e60 con 0x55563f29cb40 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9613+0+0 (secure 0 0 0) 0x7fa3dc004f30 con 0x55563f29cb40 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3d8003dd0 con 0x55563f29cb40 2024-09-16T01:20:11.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa3dc043050 con 0x55563f29cb40 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 10 client.16494 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55563f300830 con 0x55563f29cb40 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3d8003dd0 con 0x55563f29cb40 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa3dc00c040 con 0x55563f29cb40 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 10 client.16494 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 10 client.16494 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 20 client.16494 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f300ec0 0x55563f3032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 4 client.16494 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9889a58a", "release": true, "await": true}] 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.985+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9889a58a", "release": true, "await": true}) -- 0x55563f26d950 con 0x55563f300ec0 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.986+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f300ec0 0x55563f3032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.986+0000 7fa3c6ffd640 10 client.16494 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:11.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:11.986+0000 7fa3e77fe640 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f300ec0 0x55563f3032e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6469c07c2412c716 server_cookie=8a345f463fe6b58a in_seq=0 out_seq=0 2024-09-16T01:20:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3c6ffd640 1 -- 172.21.15.78:0/1245621098 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55563f26d950 con 0x55563f300ec0 2024-09-16T01:20:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3c6ffd640 10 client.16494 handle_command_reply: tid=0 2024-09-16T01:20:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 client.16494 shutdown 2024-09-16T01:20:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 msgr2=0x55563f28a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x55563f283630 tx=0x7fa3dc009ca0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 shutdown_connections 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55563f300ec0 0x55563f3032e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55563f29cb40 0x55563f28a3e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563f285650 0x55563f289ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3c6ffd640 0 client.16494 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 --2- 172.21.15.78:0/1245621098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55563f276840 0x55563f28dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.040+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 >> 172.21.15.78:0/1245621098 conn(0x55563f2725f0 msgr2=0x55563f29bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.041+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 shutdown_connections 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.041+0000 7fa3f0511740 1 -- 172.21.15.78:0/1245621098 wait complete. 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.041+0000 7fa3f0511740 20 client.16494 trim_cache size 0 max 16384 2024-09-16T01:20:12.043 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 36, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "9889a58a": { 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 36, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:20:12.044 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 142.3, 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:20:12.045 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:20:12.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.046+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 msgr2=0x7fa3d00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:12.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.046+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 0x7fa3d00a5870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00ede30 tx=0x7fa3e0047000 comp rx=0 tx=0).stop 2024-09-16T01:20:12.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.046+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 msgr2=0x7fa3e81a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:12.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.046+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0004770 tx=0x7fa3e0003c10 comp rx=0 tx=0).stop 2024-09-16T01:20:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 shutdown_connections 2024-09-16T01:20:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa3d00a33b0 0x7fa3d00a5870 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8102ef0 0x7fa3e81a4270 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3e8102570 0x7fa3e8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 --2- 172.21.15.78:0/4013386678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e8100e20 0x7fa3e81991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:12.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.047+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 >> 172.21.15.78:0/4013386678 conn(0x7fa3e80fc7d0 msgr2=0x7fa3e80ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:12.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.050+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 shutdown_connections 2024-09-16T01:20:12.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:20:12.052+0000 7fa3ee138640 1 -- 172.21.15.78:0/4013386678 wait complete. 2024-09-16T01:20:12.813 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9889a58a', total seconds elapsed: 25.9 2024-09-16T01:20:12.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:12 smithi078 ceph-mon[30937]: pgmap v292: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 11 op/s 2024-09-16T01:20:12.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:12 smithi148 ceph-mon[35030]: pgmap v292: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 11 op/s 2024-09-16T01:20:13.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:12 smithi193 ceph-mon[35701]: pgmap v292: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 11 op/s 2024-09-16T01:20:14.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:14 smithi148 ceph-mon[35030]: pgmap v293: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-09-16T01:20:15.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:14 smithi078 ceph-mon[30937]: pgmap v293: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-09-16T01:20:15.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:14 smithi193 ceph-mon[35701]: pgmap v293: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-09-16T01:20:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 540.0 s, 573.2 tps, lat 788.942 ms stddev 3479.208 2024-09-16T01:20:16.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:16 smithi148 ceph-mon[35030]: pgmap v294: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 901 KiB/s wr, 105 op/s 2024-09-16T01:20:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:16 smithi078 ceph-mon[30937]: pgmap v294: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 901 KiB/s wr, 105 op/s 2024-09-16T01:20:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:16 smithi193 ceph-mon[35701]: pgmap v294: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 901 KiB/s wr, 105 op/s 2024-09-16T01:20:18.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:18 smithi148 ceph-mon[35030]: pgmap v295: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-09-16T01:20:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:18 smithi078 ceph-mon[30937]: pgmap v295: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-09-16T01:20:19.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:18 smithi193 ceph-mon[35701]: pgmap v295: 129 pgs: 129 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-09-16T01:20:20.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:20 smithi148 ceph-mon[35030]: pgmap v296: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2024-09-16T01:20:20.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:20 smithi148 ceph-mon[35030]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T01:20:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 545.0 s, 1109.0 tps, lat 91.079 ms stddev 127.687 2024-09-16T01:20:20.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:20 smithi078 ceph-mon[30937]: pgmap v296: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2024-09-16T01:20:20.984 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:20 smithi078 ceph-mon[30937]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T01:20:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:20 smithi193 ceph-mon[35701]: pgmap v296: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2024-09-16T01:20:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:20 smithi193 ceph-mon[35701]: osdmap e102: 12 total, 12 up, 12 in 2024-09-16T01:20:22.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.757+0000 7f88493c4640 1 -- 172.21.15.78:0/713575239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8844101a30 msgr2=0x7f884410d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.757+0000 7f88493c4640 1 --2- 172.21.15.78:0/713575239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8844101a30 0x7f884410d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f883c009ca0 tx=0x7f883c02a8f0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.761+0000 7f88493c4640 1 -- 172.21.15.78:0/713575239 shutdown_connections 2024-09-16T01:20:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.761+0000 7f88493c4640 1 --2- 172.21.15.78:0/713575239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8844101a30 0x7f884410d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.761+0000 7f88493c4640 1 --2- 172.21.15.78:0/713575239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f88441014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.761+0000 7f88493c4640 1 --2- 172.21.15.78:0/713575239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88441028d0 0x7f8844102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.761+0000 7f88493c4640 1 -- 172.21.15.78:0/713575239 >> 172.21.15.78:0/713575239 conn(0x7f88440770f0 msgr2=0x7f88440fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.764+0000 7f88493c4640 1 -- 172.21.15.78:0/713575239 shutdown_connections 2024-09-16T01:20:22.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.764+0000 7f88493c4640 1 -- 172.21.15.78:0/713575239 wait complete. 2024-09-16T01:20:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.765+0000 7f88493c4640 1 Processor -- start 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.765+0000 7f88493c4640 1 -- start start 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.765+0000 7f88493c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.765+0000 7f88493c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844101a30 0x7f884419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.765+0000 7f88493c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 0x7f884419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88493c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88441120f0 con 0x7f88441028d0 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88493c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8844111f70 con 0x7f88441010b0 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59130/0 (socket says 172.21.15.78:59130) 2024-09-16T01:20:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 -- 172.21.15.78:0/3655498039 learned_addr learned my addr 172.21.15.78:0/3655498039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88427fc640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844101a30 0x7f884419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88437fe640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 0x7f884419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8844112270 con 0x7f8844101a30 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 -- 172.21.15.78:0/3655498039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844101a30 msgr2=0x7f884419ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844101a30 0x7f884419ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 -- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 msgr2=0x7f884419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 0x7f884419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 -- 172.21.15.78:0/3655498039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884419f730 con 0x7f88441010b0 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f88437fe640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 0x7f884419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.766+0000 7f8842ffd640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f883800b450 tx=0x7f883800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.767+0000 7f882bfff640 1 -- 172.21.15.78:0/3655498039 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838015020 con 0x7f88441010b0 2024-09-16T01:20:22.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.767+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884419fa20 con 0x7f88441010b0 2024-09-16T01:20:22.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.767+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88441a7330 con 0x7f88441010b0 2024-09-16T01:20:22.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.767+0000 7f882bfff640 1 -- 172.21.15.78:0/3655498039 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8838011d70 con 0x7f88441010b0 2024-09-16T01:20:22.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.767+0000 7f882bfff640 1 -- 172.21.15.78:0/3655498039 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88380103e0 con 0x7f88441010b0 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.769+0000 7f884c240740 1 Processor -- start 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.769+0000 7f884c240740 1 -- start start 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.769+0000 7f884c240740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f884c240740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f884c240740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c8299ecb0 0x559c829a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f884c240740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559c828f82f0 con 0x559c82993af0 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f884c240740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559c828a0270 con 0x559c82994410 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f884c240740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559c828f42b0 con 0x559c8299ecb0 2024-09-16T01:20:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35006/0 (socket says 172.21.15.78:35006) 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 -- 172.21.15.78:0/1850771637 learned_addr learned my addr 172.21.15.78:0/1850771637 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88427fc640 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c8299ecb0 0x559c829a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 -- 172.21.15.78:0/1850771637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c8299ecb0 msgr2=0x559c829a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c8299ecb0 0x559c829a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 -- 172.21.15.78:0/1850771637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 msgr2=0x559c8299e770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 -- 172.21.15.78:0/1850771637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c8289f340 con 0x559c82993af0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.770+0000 7f88437fe640 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f883c02aee0 tx=0x7f883c02f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00e98cb5c6b0f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f88297fa640 1 -- 172.21.15.78:0/1850771637 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c003c70 con 0x559c82993af0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f88297fa640 1 -- 172.21.15.78:0/1850771637 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f883c036040 con 0x559c82993af0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f88297fa640 1 -- 172.21.15.78:0/1850771637 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c0047f0 con 0x559c82993af0 2024-09-16T01:20:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f882bfff640 1 -- 172.21.15.78:0/3655498039 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f8838018020 con 0x7f88441010b0 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f884c240740 1 -- 172.21.15.78:0/1850771637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 msgr2=0x559c82993ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f884c240740 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f883c02aee0 tx=0x7f883c02f4c0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f882bfff640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 0x7f882c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f884c240740 1 -- 172.21.15.78:0/1850771637 shutdown_connections 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f884c240740 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c8299ecb0 0x559c829a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.771+0000 7f884c240740 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 --2- 172.21.15.78:0/1850771637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c82993af0 0x559c82993ed0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 -- 172.21.15.78:0/1850771637 >> 172.21.15.78:0/1850771637 conn(0x559c82990060 msgr2=0x559c82990440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f88437fe640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 0x7f882c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f882bfff640 1 -- 172.21.15.78:0/3655498039 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f88380cf960 con 0x7f88441010b0 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 -- 172.21.15.78:0/1850771637 shutdown_connections 2024-09-16T01:20:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 -- 172.21.15.78:0/1850771637 wait complete. 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f88437fe640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 0x7f882c0a5920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f883c0044d0 tx=0x7f883c004990 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 -- start start 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 Processor -- start 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.772+0000 7f884c240740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559c829a1880 con 0x559c8299ecb0 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559c82881290 con 0x559c82994410 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559c829a1a00 con 0x559c82993af0 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 10 client.0 _ll_register_callbacks cb 0x559c828b2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35016/0 (socket says 172.21.15.78:35016) 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 -- 172.21.15.78:0/1087355958 learned_addr learned my addr 172.21.15.78:0/1087355958 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88427fc640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 -- 172.21.15.78:0/1087355958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 msgr2=0x559c8299ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 -- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 msgr2=0x559c8299bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 -- 172.21.15.78:0/1087355958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c8260b210 con 0x559c8299ecb0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88427fc640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f88437fe640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f883c004250 tx=0x7f883c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f1b421710ce806 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f8826ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c0317d0 con 0x559c8299ecb0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f883c031970 con 0x559c8299ecb0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.773+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f883c036040 con 0x559c8299ecb0 2024-09-16T01:20:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f883c068050 con 0x559c8299ecb0 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 10 client.16506 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559c826d7620 con 0x559c8299ecb0 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f883c036040 con 0x559c8299ecb0 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f883c04f050 con 0x559c8299ecb0 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 10 client.16506 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 20 client.16506 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c829b7220 0x559c829b9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 4 client.16506 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c82905a20 con 0x559c829b7220 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.774+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c829b7220 0x559c829b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.775+0000 7f8826ff5640 10 client.16506 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.775+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c829b7220 0x559c829b9610 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d41617f49cb818d server_cookie=2c6240451b4a93c5 in_seq=0 out_seq=0 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f8826ff5640 1 -- 172.21.15.78:0/1087355958 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559c82905a20 con 0x559c829b7220 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f8826ff5640 10 client.16506 handle_command_reply: tid=0 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 client.16506 shutdown 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 msgr2=0x559c8299c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f883c004250 tx=0x7f883c0095c0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 shutdown_connections 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c829b7220 0x559c829b9610 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c8299c500 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f8826ff5640 0 client.16506 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c8299bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 --2- 172.21.15.78:0/1087355958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 >> 172.21.15.78:0/1087355958 conn(0x559c82990060 msgr2=0x559c82995dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 shutdown_connections 2024-09-16T01:20:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 1 -- 172.21.15.78:0/1087355958 wait complete. 2024-09-16T01:20:22.784 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.776+0000 7f884c240740 20 client.16506 trim_cache size 0 max 16384 2024-09-16T01:20:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 Processor -- start 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 -- start start 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829bed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88300128d0 con 0x559c8299ecb0 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88300101a0 con 0x559c82994410 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f884c240740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f883000e010 con 0x559c82993af0 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35020/0 (socket says 172.21.15.78:35020) 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 -- 172.21.15.78:0/3545923261 learned_addr learned my addr 172.21.15.78:0/3545923261 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88427fc640 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829bed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f8842ffd640 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 -- 172.21.15.78:0/3545923261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 msgr2=0x559c8299c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 -- 172.21.15.78:0/3545923261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 msgr2=0x559c829bed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829bed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 -- 172.21.15.78:0/3545923261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c82994240 con 0x559c8299ecb0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f88437fe640 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f883c006010 tx=0x7f883c005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9dc6cd20586b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f8828ff9640 1 -- 172.21.15.78:0/3545923261 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c030da0 con 0x559c8299ecb0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f8828ff9640 1 -- 172.21.15.78:0/3545923261 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f883c036070 con 0x559c8299ecb0 2024-09-16T01:20:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.791+0000 7f8828ff9640 1 -- 172.21.15.78:0/3545923261 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c02fba0 con 0x559c8299ecb0 2024-09-16T01:20:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- 172.21.15.78:0/3545923261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 msgr2=0x559c829bf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f883c006010 tx=0x7f883c005b70 comp rx=0 tx=0).stop 2024-09-16T01:20:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- 172.21.15.78:0/3545923261 shutdown_connections 2024-09-16T01:20:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829bf2c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829bed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 --2- 172.21.15.78:0/3545923261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c8299c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- 172.21.15.78:0/3545923261 >> 172.21.15.78:0/3545923261 conn(0x559c82990060 msgr2=0x559c8299db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- 172.21.15.78:0/3545923261 shutdown_connections 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- 172.21.15.78:0/3545923261 wait complete. 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.792+0000 7f884c240740 1 -- start start 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 Processor -- start 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c82999780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f883000e660 con 0x559c8299ecb0 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f883000fad0 con 0x559c82994410 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f883000ed30 con 0x559c82993af0 2024-09-16T01:20:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f884c240740 10 client.0 _ll_register_callbacks cb 0x7f883c02f410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35026/0 (socket says 172.21.15.78:35026) 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 -- 172.21.15.78:0/1134501422 learned_addr learned my addr 172.21.15.78:0/1134501422 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c82999780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 -- 172.21.15.78:0/1134501422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 msgr2=0x559c82999780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c82999780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 -- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 msgr2=0x559c829af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 -- 172.21.15.78:0/1134501422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c829a0010 con 0x559c8299ecb0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c82999780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f88437fe640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f883c000f80 tx=0x7f883c02f6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d361cb7ca568f0cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f8826ff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c0319e0 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.793+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f883c0307f0 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f883c036070 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f883c07d050 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 10 client.16512 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559c82a22110 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f883c036070 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f883c044070 con 0x559c8299ecb0 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 10 client.16512 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 20 client.16512 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c82a22760 0x559c82a24b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 4 client.16512 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:20:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.794+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c828ffd70 con 0x559c82a22760 2024-09-16T01:20:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.795+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c82a22760 0x559c82a24b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.795+0000 7f8826ff5640 10 client.16512 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:22.795+0000 7f8842ffd640 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c82a22760 0x559c82a24b50 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f785373eb7cb053e server_cookie=f60ae3ea31caafcd in_seq=0 out_seq=0 2024-09-16T01:20:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:22 smithi148 ceph-mon[35030]: pgmap v298: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 4.3 MiB/s wr, 489 op/s 2024-09-16T01:20:23.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:22 smithi078 ceph-mon[30937]: pgmap v298: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 4.3 MiB/s wr, 489 op/s 2024-09-16T01:20:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:22 smithi193 ceph-mon[35701]: pgmap v298: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 4.3 MiB/s wr, 489 op/s 2024-09-16T01:20:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.211+0000 7f8826ff5640 1 -- 172.21.15.78:0/1134501422 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c828ffd70 con 0x559c82a22760 2024-09-16T01:20:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.211+0000 7f8826ff5640 10 client.16512 handle_command_reply: tid=0 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.211+0000 7f884c240740 1 client.16512 shutdown 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 msgr2=0x559c829afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f883c000f80 tx=0x7f883c02f6e0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 shutdown_connections 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559c82a22760 0x559c82a24b50 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c8299ecb0 0x559c829afd40 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f8826ff5640 0 client.16512 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559c82994410 0x559c829af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 --2- 172.21.15.78:0/1134501422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c82993af0 0x559c82999780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 >> 172.21.15.78:0/1134501422 conn(0x559c82990060 msgr2=0x559c82995550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 shutdown_connections 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.212+0000 7f884c240740 1 -- 172.21.15.78:0/1134501422 wait complete. 2024-09-16T01:20:23.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.213+0000 7f884c240740 20 client.16512 trim_cache size 0 max 16384 2024-09-16T01:20:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:20:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:20:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:20:23.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.213+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 msgr2=0x7f882c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.213+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 0x7f882c0a5920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f883c0044d0 tx=0x7f883c004990 comp rx=0 tx=0).stop 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.213+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 msgr2=0x7f884419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.213+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f883800b450 tx=0x7f883800b910 comp rx=0 tx=0).stop 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 shutdown_connections 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f882c0a3460 0x7f882c0a5920 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88441028d0 0x7f884419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844101a30 0x7f884419ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 --2- 172.21.15.78:0/3655498039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88441010b0 0x7f884419a740 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 >> 172.21.15.78:0/3655498039 conn(0x7f88440770f0 msgr2=0x7f8844104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 shutdown_connections 2024-09-16T01:20:23.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:23.214+0000 7f88493c4640 1 -- 172.21.15.78:0/3655498039 wait complete. 2024-09-16T01:20:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:24 smithi148 ceph-mon[35030]: pgmap v299: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-09-16T01:20:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:24 smithi078 ceph-mon[30937]: pgmap v299: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-09-16T01:20:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:24 smithi193 ceph-mon[35701]: pgmap v299: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-09-16T01:20:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 550.0 s, 606.2 tps, lat 163.773 ms stddev 219.935 2024-09-16T01:20:26.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:26 smithi148 ceph-mon[35030]: pgmap v300: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 7.1 MiB/s wr, 805 op/s 2024-09-16T01:20:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:26 smithi078 ceph-mon[30937]: pgmap v300: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 7.1 MiB/s wr, 805 op/s 2024-09-16T01:20:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:26 smithi193 ceph-mon[35701]: pgmap v300: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 7.1 MiB/s wr, 805 op/s 2024-09-16T01:20:28.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:28 smithi148 ceph-mon[35030]: pgmap v301: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.9 MiB/s wr, 879 op/s 2024-09-16T01:20:29.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:28 smithi078 ceph-mon[30937]: pgmap v301: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.9 MiB/s wr, 879 op/s 2024-09-16T01:20:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:28 smithi193 ceph-mon[35701]: pgmap v301: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 7.9 MiB/s wr, 879 op/s 2024-09-16T01:20:30.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:30 smithi148 ceph-mon[35030]: pgmap v302: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.2 MiB/s wr, 822 op/s 2024-09-16T01:20:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 555.0 s, 849.0 tps, lat 116.466 ms stddev 130.640 2024-09-16T01:20:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:30 smithi078 ceph-mon[30937]: pgmap v302: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.2 MiB/s wr, 822 op/s 2024-09-16T01:20:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:30 smithi193 ceph-mon[35701]: pgmap v302: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.2 MiB/s wr, 822 op/s 2024-09-16T01:20:32.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:32 smithi148 ceph-mon[35030]: pgmap v303: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.1 MiB/s wr, 798 op/s 2024-09-16T01:20:33.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:32 smithi078 ceph-mon[30937]: pgmap v303: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.1 MiB/s wr, 798 op/s 2024-09-16T01:20:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:32 smithi193 ceph-mon[35701]: pgmap v303: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.1 MiB/s wr, 798 op/s 2024-09-16T01:20:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:34 smithi148 ceph-mon[35030]: pgmap v304: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 670 op/s 2024-09-16T01:20:35.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:34 smithi078 ceph-mon[30937]: pgmap v304: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 670 op/s 2024-09-16T01:20:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:34 smithi193 ceph-mon[35701]: pgmap v304: 129 pgs: 129 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 670 op/s 2024-09-16T01:20:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 560.0 s, 855.8 tps, lat 115.850 ms stddev 136.972 2024-09-16T01:20:36.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.351+0000 7fb90a9f8640 1 -- 172.21.15.78:0/2049785280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041069c0 msgr2=0x7fb904106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/2049785280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041069c0 0x7fb904106da0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009ca0 tx=0x7fb8f402a8b0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 -- 172.21.15.78:0/2049785280 shutdown_connections 2024-09-16T01:20:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/2049785280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904101350 0x7fb90410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.355 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/2049785280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9041009b0 0x7fb904100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/2049785280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041069c0 0x7fb904106da0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 -- 172.21.15.78:0/2049785280 >> 172.21.15.78:0/2049785280 conn(0x7fb904077990 msgr2=0x7fb9040fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 -- 172.21.15.78:0/2049785280 shutdown_connections 2024-09-16T01:20:36.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.352+0000 7fb90a9f8640 1 -- 172.21.15.78:0/2049785280 wait complete. 2024-09-16T01:20:36.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.354+0000 7fb90a9f8640 1 Processor -- start 2024-09-16T01:20:36.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.359+0000 7fb90a9f8640 1 -- start start 2024-09-16T01:20:36.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041009b0 0x7fb9040fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 0x7fb9040fd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 0x7fb9040fd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb90406c620 con 0x7fb904101350 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb90406c4a0 con 0x7fb9041009b0 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb90a9f8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb90406c7a0 con 0x7fb9041069c0 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 0x7fb9040fd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 0x7fb9040fd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56214/0 (socket says 172.21.15.78:56214) 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb9037fe640 1 -- 172.21.15.78:0/1413517615 learned_addr learned my addr 172.21.15.78:0/1413517615 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:36.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.360+0000 7fb908f95640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 0x7fb9040fd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 -- 172.21.15.78:0/1413517615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041009b0 msgr2=0x7fb9040fcd10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041009b0 0x7fb9040fcd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 -- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 msgr2=0x7fb9040fd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 0x7fb9040fd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 -- 172.21.15.78:0/1413517615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb904072f10 con 0x7fb9041069c0 2024-09-16T01:20:36.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb908f95640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 0x7fb9040fd790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00e970 tx=0x7fb8fc00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.361+0000 7fb9017fa640 1 -- 172.21.15.78:0/1413517615 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc00ccc0 con 0x7fb9041069c0 2024-09-16T01:20:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.362+0000 7fb9017fa640 1 -- 172.21.15.78:0/1413517615 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8fc005020 con 0x7fb9041069c0 2024-09-16T01:20:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.362+0000 7fb9017fa640 1 -- 172.21.15.78:0/1413517615 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc004a00 con 0x7fb9041069c0 2024-09-16T01:20:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.362+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb904073200 con 0x7fb9041069c0 2024-09-16T01:20:36.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.362+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb904073700 con 0x7fb9041069c0 2024-09-16T01:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.365+0000 7fb9017fa640 1 -- 172.21.15.78:0/1413517615 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb8fc004ba0 con 0x7fb9041069c0 2024-09-16T01:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 Processor -- start 2024-09-16T01:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb9017fa640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 0x7fb8f00a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 -- start start 2024-09-16T01:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb9017fa640 1 -- 172.21.15.78:0/1413517615 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb8fc0d0010 con 0x7fb9041069c0 2024-09-16T01:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dd4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dd4cb0 0x564099dd70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564099d2e2f0 con 0x564099dd4cb0 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564099cd6270 con 0x564099dca410 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb90d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564099d2a2b0 con 0x564099dc9af0 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb9037fe640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 0x7fb8f00a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb903fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb903fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:49186/0 (socket says 172.21.15.78:49186) 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.366+0000 7fb903fff640 1 -- 172.21.15.78:0/2134773512 learned_addr learned my addr 172.21.15.78:0/2134773512 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 -- 172.21.15.78:0/2134773512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 msgr2=0x564099dd4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dd4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 -- 172.21.15.78:0/2134773512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dd4cb0 msgr2=0x564099dd70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dd4cb0 0x564099dd70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 -- 172.21.15.78:0/2134773512 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564099cd5340 con 0x564099dc9af0 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb903fff640 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4005e30 tx=0x7fb8f4009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=374e96d2b3e64e06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb9037fe640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 0x7fb8f00a5b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9040fe430 tx=0x7fb8ec0095c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/2134773512 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40303a0 con 0x564099dc9af0 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/2134773512 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f403f070 con 0x564099dc9af0 2024-09-16T01:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.367+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/2134773512 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4030e50 con 0x564099dc9af0 2024-09-16T01:20:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- 172.21.15.78:0/2134773512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 msgr2=0x564099dc9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4005e30 tx=0x7fb8f4009a20 comp rx=0 tx=0).stop 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- 172.21.15.78:0/2134773512 shutdown_connections 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dd4cb0 0x564099dd70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dd4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 --2- 172.21.15.78:0/2134773512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dc9ed0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- 172.21.15.78:0/2134773512 >> 172.21.15.78:0/2134773512 conn(0x564099dc6060 msgr2=0x564099dc6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- 172.21.15.78:0/2134773512 shutdown_connections 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- 172.21.15.78:0/2134773512 wait complete. 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.368+0000 7fb90d866740 1 -- start start 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 Processor -- start 2024-09-16T01:20:36.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 0x564099dd2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099dd25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x564099dd7880 con 0x564099dca410 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564099cb7290 con 0x564099dd4cb0 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564099dd7a00 con 0x564099dc9af0 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb90d866740 10 client.0 _ll_register_callbacks cb 0x564099ce8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb903fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb903fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:49202/0 (socket says 172.21.15.78:49202) 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 0x564099dd2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb903fff640 1 -- 172.21.15.78:0/3848568611 learned_addr learned my addr 172.21.15.78:0/3848568611 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:36.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb908f95640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099dd25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.369+0000 7fb9037fe640 1 -- 172.21.15.78:0/3848568611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 msgr2=0x564099dd1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb9037fe640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb9037fe640 1 -- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 msgr2=0x564099dd25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb9037fe640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099dd25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb9037fe640 1 -- 172.21.15.78:0/3848568611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564099a41210 con 0x564099dca410 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb903fff640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb9037fe640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 0x564099dd2060 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec014710 tx=0x7fb8ec0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16fc430fbb68c5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb8e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec024020 con 0x564099dca410 2024-09-16T01:20:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8f403f070 con 0x564099dca410 2024-09-16T01:20:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.370+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb8ec016700 con 0x564099dca410 2024-09-16T01:20:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8ec024020 con 0x564099dca410 2024-09-16T01:20:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 10 client.16518 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564099b0d620 con 0x564099dca410 2024-09-16T01:20:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f403f070 con 0x564099dca410 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb8ec01f030 con 0x564099dca410 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 10 client.16518 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 20 client.16518 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099de4d30 0x564099de7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 4 client.16518 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.371+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564099d3ba20 con 0x564099de4d30 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.372+0000 7fb903fff640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099de4d30 0x564099de7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.373+0000 7fb8e37fe640 10 client.16518 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.373+0000 7fb903fff640 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099de4d30 0x564099de7140 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec76de92ae451289 server_cookie=ad9b136889c6a09 in_seq=0 out_seq=0 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb8e37fe640 1 -- 172.21.15.78:0/3848568611 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564099d3ba20 con 0x564099de4d30 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb8e37fe640 10 client.16518 handle_command_reply: tid=0 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 client.16518 shutdown 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 msgr2=0x564099dd2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 0x564099dd2060 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec014710 tx=0x7fb8ec0047d0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 shutdown_connections 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099de4d30 0x564099de7140 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099dd25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dca410 0x564099dd2060 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 --2- 172.21.15.78:0/3848568611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dc9af0 0x564099dd1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 >> 172.21.15.78:0/3848568611 conn(0x564099dc6060 msgr2=0x564099dcbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb8e37fe640 0 client.16518 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 shutdown_connections 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.374+0000 7fb90d866740 1 -- 172.21.15.78:0/3848568611 wait complete. 2024-09-16T01:20:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.375+0000 7fb90d866740 20 client.16518 trim_cache size 0 max 16384 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.398+0000 7fb90d866740 1 Processor -- start 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.398+0000 7fb90d866740 1 -- start start 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099dd23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 0x564099dd3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099deca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e4012960 con 0x564099dc9af0 2024-09-16T01:20:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e4010230 con 0x564099dd4cb0 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb90d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e400e0a0 con 0x564099dca410 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099dd23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099dd23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56242/0 (socket says 172.21.15.78:56242) 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb903fff640 1 -- 172.21.15.78:0/4279264798 learned_addr learned my addr 172.21.15.78:0/4279264798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 0x564099dd3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb908f95640 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099deca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 -- 172.21.15.78:0/4279264798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 msgr2=0x564099deca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099deca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 -- 172.21.15.78:0/4279264798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 msgr2=0x564099dd23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099dd23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.399+0000 7fb9037fe640 1 -- 172.21.15.78:0/4279264798 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564099dc5e90 con 0x564099dca410 2024-09-16T01:20:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.400+0000 7fb9037fe640 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 0x564099dd3a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec02b040 tx=0x7fb8ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=187e5ed7bf104d37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.400+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/4279264798 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec01c040 con 0x564099dca410 2024-09-16T01:20:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.400+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/4279264798 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ec0208d0 con 0x564099dca410 2024-09-16T01:20:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.400+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/4279264798 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec016bd0 con 0x564099dca410 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- 172.21.15.78:0/4279264798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 msgr2=0x564099dd3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 0x564099dd3a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec02b040 tx=0x7fb8ec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- 172.21.15.78:0/4279264798 shutdown_connections 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dd4cb0 0x564099deca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dca410 0x564099dd3a90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 --2- 172.21.15.78:0/4279264798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099dd23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- 172.21.15.78:0/4279264798 >> 172.21.15.78:0/4279264798 conn(0x564099dc6060 msgr2=0x564099dd3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- 172.21.15.78:0/4279264798 shutdown_connections 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- 172.21.15.78:0/4279264798 wait complete. 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 -- start start 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.401+0000 7fb90d866740 1 Processor -- start 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099ddd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 0x564099dceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e400e6f0 con 0x564099dc9af0 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e400fb60 con 0x564099dca410 2024-09-16T01:20:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e400edc0 con 0x564099dd4cb0 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099ddd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb90d866740 10 client.0 _ll_register_callbacks cb 0x7fb8ec009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099ddd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56258/0 (socket says 172.21.15.78:56258) 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb908f95640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 0x564099dceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb903fff640 1 -- 172.21.15.78:0/2542272354 learned_addr learned my addr 172.21.15.78:0/2542272354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 -- 172.21.15.78:0/2542272354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 msgr2=0x564099dceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 0x564099dceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 -- 172.21.15.78:0/2542272354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 msgr2=0x564099ddd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099ddd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 -- 172.21.15.78:0/2542272354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564099dcf690 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb908f95640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 0x564099dceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb9037fe640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dce890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x564099dc5a20 tx=0x7fb8ec016990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e202f300d737399c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.402+0000 7fb8e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec041020 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fb8ec028560 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8ec0208d0 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8ec04c050 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 10 client.25862 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564099e4fa80 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ec0208d0 con 0x564099dca410 2024-09-16T01:20:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb8ec01f040 con 0x564099dca410 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 10 client.25862 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 10 client.25862 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 20 client.25862 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099e500e0 0x564099e524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.403+0000 7fb90d866740 4 client.25862 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.404+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564099d35d70 con 0x564099e500e0 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.404+0000 7fb903fff640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099e500e0 0x564099e524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.404+0000 7fb8e37fe640 10 client.25862 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.404+0000 7fb903fff640 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099e500e0 0x564099e524d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e7da9c67e360495 server_cookie=8c1c0265d5553ebf in_seq=0 out_seq=0 2024-09-16T01:20:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.416+0000 7fb8e37fe640 1 -- 172.21.15.78:0/2542272354 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564099d35d70 con 0x564099e500e0 2024-09-16T01:20:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.416+0000 7fb8e37fe640 10 client.25862 handle_command_reply: tid=0 2024-09-16T01:20:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.416+0000 7fb90d866740 1 client.25862 shutdown 2024-09-16T01:20:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.416+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 msgr2=0x564099dce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.416+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dce890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x564099dc5a20 tx=0x7fb8ec016990 comp rx=0 tx=0).stop 2024-09-16T01:20:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 shutdown_connections 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x564099e500e0 0x564099e524d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564099dd4cb0 0x564099dceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564099dca410 0x564099dce890 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 --2- 172.21.15.78:0/2542272354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564099dc9af0 0x564099ddd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb8e37fe640 0 client.25862 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 >> 172.21.15.78:0/2542272354 conn(0x564099dc6060 msgr2=0x564099dd3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 shutdown_connections 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 1 -- 172.21.15.78:0/2542272354 wait complete. 2024-09-16T01:20:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90d866740 20 client.25862 trim_cache size 0 max 16384 2024-09-16T01:20:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:20:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:20:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:20:36.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 msgr2=0x7fb8f00a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 0x7fb8f00a5b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9040fe430 tx=0x7fb8ec0095c0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 msgr2=0x7fb9040fd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 0x7fb9040fd790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00e970 tx=0x7fb8fc00ee30 comp rx=0 tx=0).stop 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 shutdown_connections 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a3640 0x7fb8f00a5b00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9041069c0 0x7fb9040fd790 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904101350 0x7fb9040fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 --2- 172.21.15.78:0/1413517615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041009b0 0x7fb9040fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 >> 172.21.15.78:0/1413517615 conn(0x7fb904077990 msgr2=0x7fb904069a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 shutdown_connections 2024-09-16T01:20:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:36.417+0000 7fb90a9f8640 1 -- 172.21.15.78:0/1413517615 wait complete. 2024-09-16T01:20:36.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:36 smithi148 ceph-mon[35030]: pgmap v305: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 857 op/s 2024-09-16T01:20:37.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:36 smithi078 ceph-mon[30937]: pgmap v305: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 857 op/s 2024-09-16T01:20:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:36 smithi193 ceph-mon[35701]: pgmap v305: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 857 op/s 2024-09-16T01:20:38.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:38 smithi148 ceph-mon[35030]: pgmap v306: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-09-16T01:20:39.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:38 smithi078 ceph-mon[30937]: pgmap v306: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-09-16T01:20:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:38 smithi193 ceph-mon[35701]: pgmap v306: 129 pgs: 129 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-09-16T01:20:40.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:40 smithi148 ceph-mon[35030]: pgmap v307: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-16T01:20:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 565.0 s, 661.4 tps, lat 150.996 ms stddev 181.175 2024-09-16T01:20:41.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:40 smithi078 ceph-mon[30937]: pgmap v307: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-16T01:20:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:40 smithi193 ceph-mon[35701]: pgmap v307: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-16T01:20:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:42 smithi148 ceph-mon[35030]: pgmap v308: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.4 MiB/s wr, 596 op/s 2024-09-16T01:20:43.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:42 smithi078 ceph-mon[30937]: pgmap v308: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.4 MiB/s wr, 596 op/s 2024-09-16T01:20:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:42 smithi193 ceph-mon[35701]: pgmap v308: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.4 MiB/s wr, 596 op/s 2024-09-16T01:20:45.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:44 smithi078 ceph-mon[30937]: pgmap v309: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-09-16T01:20:45.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:44 smithi193 ceph-mon[35701]: pgmap v309: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-09-16T01:20:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:44 smithi148 ceph-mon[35030]: pgmap v309: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.3 MiB/s wr, 468 op/s 2024-09-16T01:20:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 570.0 s, 708.6 tps, lat 143.647 ms stddev 161.818 2024-09-16T01:20:47.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:46 smithi078 ceph-mon[30937]: pgmap v310: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 624 op/s 2024-09-16T01:20:47.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:46 smithi193 ceph-mon[35701]: pgmap v310: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 624 op/s 2024-09-16T01:20:47.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:46 smithi148 ceph-mon[35030]: pgmap v310: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 624 op/s 2024-09-16T01:20:49.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:48 smithi078 ceph-mon[30937]: pgmap v311: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-09-16T01:20:49.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:48 smithi193 ceph-mon[35701]: pgmap v311: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-09-16T01:20:49.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:48 smithi148 ceph-mon[35030]: pgmap v311: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-09-16T01:20:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.560+0000 7f8eca2c0640 1 -- 172.21.15.78:0/952449852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 msgr2=0x7f8ec4100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.560+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/952449852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec4100870 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400aa40 tx=0x7f8eb402b760 comp rx=0 tx=0).stop 2024-09-16T01:20:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.562+0000 7f8eca2c0640 1 -- 172.21.15.78:0/952449852 shutdown_connections 2024-09-16T01:20:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.562+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/952449852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ec41026f0 0x7f8ec410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.562+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/952449852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec4101d70 0x7f8ec41021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.562+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/952449852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec4100870 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.562+0000 7f8eca2c0640 1 -- 172.21.15.78:0/952449852 >> 172.21.15.78:0/952449852 conn(0x7f8ec40770f0 msgr2=0x7f8ec40fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.569+0000 7f8eca2c0640 1 -- 172.21.15.78:0/952449852 shutdown_connections 2024-09-16T01:20:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.573+0000 7f8eca2c0640 1 -- 172.21.15.78:0/952449852 wait complete. 2024-09-16T01:20:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.573+0000 7f8eca2c0640 1 Processor -- start 2024-09-16T01:20:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8eca2c0640 1 -- start start 2024-09-16T01:20:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8eca2c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8eca2c0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ec4101d70 0x7f8ec41966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8eca2c0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec41026f0 0x7f8ec419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.575+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32956/0 (socket says 172.21.15.78:32956) 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.575+0000 7f8ec3fff640 1 -- 172.21.15.78:0/1772463706 learned_addr learned my addr 172.21.15.78:0/1772463706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.578+0000 7f8ec885d640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec41026f0 0x7f8ec419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.574+0000 7f8eca2c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410fe20 con 0x7f8ec4100490 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.578+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410fca0 con 0x7f8ec41026f0 2024-09-16T01:20:49.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.578+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410ffa0 con 0x7f8ec4101d70 2024-09-16T01:20:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.580+0000 7f8ec3fff640 1 -- 172.21.15.78:0/1772463706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ec4101d70 msgr2=0x7f8ec41966e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:20:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.580+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ec4101d70 0x7f8ec41966e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.580+0000 7f8ec3fff640 1 -- 172.21.15.78:0/1772463706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec41026f0 msgr2=0x7f8ec419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.580+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec41026f0 0x7f8ec419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.580+0000 7f8ec3fff640 1 -- 172.21.15.78:0/1772463706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec419b190 con 0x7f8ec4100490 2024-09-16T01:20:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.581+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402bc80 tx=0x7f8eb4030bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.583+0000 7f8ec1ffb640 1 -- 172.21.15.78:0/1772463706 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb403f070 con 0x7f8ec4100490 2024-09-16T01:20:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.583+0000 7f8ec1ffb640 1 -- 172.21.15.78:0/1772463706 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb4003cd0 con 0x7f8ec4100490 2024-09-16T01:20:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.583+0000 7f8ec1ffb640 1 -- 172.21.15.78:0/1772463706 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4036040 con 0x7f8ec4100490 2024-09-16T01:20:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.585+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec419b420 con 0x7f8ec4100490 2024-09-16T01:20:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.585+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec419b7d0 con 0x7f8ec4100490 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 Processor -- start 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 -- start start 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c53d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560b57bb82f0 con 0x560b57c539b0 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560b57b60270 con 0x560b57c6fcc0 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ecd054740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b57bb42b0 con 0x560b57c542d0 2024-09-16T01:20:49.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.590+0000 7f8ec1ffb640 1 -- 172.21.15.78:0/1772463706 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f8eb4042070 con 0x7f8ec4100490 2024-09-16T01:20:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.592+0000 7f8ec885d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c53d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.592+0000 7f8ec885d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c53d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32958/0 (socket says 172.21.15.78:32958) 2024-09-16T01:20:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.592+0000 7f8ec885d640 1 -- 172.21.15.78:0/2073772114 learned_addr learned my addr 172.21.15.78:0/2073772114 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.592+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.594+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.595+0000 7f8ec1ffb640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 0x7f8ea40a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.595+0000 7f8ec1ffb640 1 -- 172.21.15.78:0/1772463706 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f8eb40f06b0 con 0x7f8ec4100490 2024-09-16T01:20:49.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec3fff640 1 -- 172.21.15.78:0/2073772114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 msgr2=0x560b57c720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec3fff640 1 -- 172.21.15.78:0/2073772114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 msgr2=0x560b57c53d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c53d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec3fff640 1 -- 172.21.15.78:0/2073772114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b57b5f340 con 0x560b57c542d0 2024-09-16T01:20:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.596+0000 7f8ec885d640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 0x7f8ea40a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.597+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6f780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40ed9b0 tx=0x7f8eb4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99114e1c19fb50a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.598+0000 7f8ec885d640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 0x7f8ea40a9f20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc014bb0 tx=0x7f8ebc004ab0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.600+0000 7f8eba7fc640 1 -- 172.21.15.78:0/2073772114 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40bc310 con 0x560b57c542d0 2024-09-16T01:20:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.600+0000 7f8eba7fc640 1 -- 172.21.15.78:0/2073772114 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb40dc070 con 0x560b57c542d0 2024-09-16T01:20:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.600+0000 7f8eba7fc640 1 -- 172.21.15.78:0/2073772114 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40bbea0 con 0x560b57c542d0 2024-09-16T01:20:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.601+0000 7f8ecd054740 1 -- 172.21.15.78:0/2073772114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 msgr2=0x560b57c6f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.601+0000 7f8ecd054740 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6f780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40ed9b0 tx=0x7f8eb4000bc0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.603+0000 7f8ecd054740 1 -- 172.21.15.78:0/2073772114 shutdown_connections 2024-09-16T01:20:49.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.603+0000 7f8ecd054740 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.603+0000 7f8ecd054740 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6f780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.603+0000 7f8ecd054740 1 --2- 172.21.15.78:0/2073772114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c53d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.603+0000 7f8ecd054740 1 -- 172.21.15.78:0/2073772114 >> 172.21.15.78:0/2073772114 conn(0x560b57c50060 msgr2=0x560b57c50440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.604+0000 7f8ecd054740 1 -- 172.21.15.78:0/2073772114 shutdown_connections 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.606+0000 7f8ecd054740 1 -- 172.21.15.78:0/2073772114 wait complete. 2024-09-16T01:20:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.606+0000 7f8ecd054740 1 -- start start 2024-09-16T01:20:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.607+0000 7f8ecd054740 1 Processor -- start 2024-09-16T01:20:49.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.607+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c6d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ebbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ebbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:44122/0 (socket says 172.21.15.78:44122) 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ebbfff640 1 -- 172.21.15.78:0/1722264612 learned_addr learned my addr 172.21.15.78:0/1722264612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560b57c72650 con 0x560b57c539b0 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560b57b41290 con 0x560b57c6fcc0 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.608+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560b57c727d0 con 0x560b57c542d0 2024-09-16T01:20:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ecd054740 10 client.0 _ll_register_callbacks cb 0x560b57b72140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 -- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 msgr2=0x560b57c6d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c6d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 -- 172.21.15.78:0/1722264612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 msgr2=0x560b57c6cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b578cb210 con 0x560b57c542d0 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8eac011480 tx=0x7f8eac011940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced44b6eebfaffbb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.611 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.609+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb40dc070 con 0x560b57c542d0 2024-09-16T01:20:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.612+0000 7f8eb37fe640 10 client.25921 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:20:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.612+0000 7f8eb37fe640 1 -- 172.21.15.78:0/1722264612 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac012980 con 0x560b57c542d0 2024-09-16T01:20:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.612+0000 7f8eb37fe640 1 -- 172.21.15.78:0/1722264612 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f8eac012b20 con 0x560b57c542d0 2024-09-16T01:20:49.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.612+0000 7f8eb37fe640 1 -- 172.21.15.78:0/1722264612 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8eac025050 con 0x560b57c542d0 2024-09-16T01:20:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.613+0000 7f8ecd054740 10 client.25921 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:49.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.613+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x560b57997620 con 0x560b57c542d0 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.617+0000 7f8eb37fe640 1 -- 172.21.15.78:0/1722264612 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8eac00f440 con 0x560b57c542d0 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 10 client.25921 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 10 client.25921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 20 client.25921 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57c7f8c0 0x560b57c81cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 4 client.25921 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8eb40dc070 con 0x560b57c7f8c0 2024-09-16T01:20:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.618+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57c7f8c0 0x560b57c81cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.620+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57c7f8c0 0x560b57c81cf0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5582b6b29e3c46 server_cookie=c8a57c75ae8978c4 in_seq=0 out_seq=0 2024-09-16T01:20:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.620+0000 7f8eb37fe640 10 client.25921 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.620+0000 7f8eb37fe640 1 -- 172.21.15.78:0/1722264612 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f8eb40dc070 con 0x560b57c7f8c0 2024-09-16T01:20:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.620+0000 7f8eb37fe640 10 client.25921 handle_command_reply: tid=0 2024-09-16T01:20:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.621+0000 7f8ecd054740 1 client.25921 shutdown 2024-09-16T01:20:49.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.623+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 msgr2=0x560b57c6d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.623+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8eac011480 tx=0x7f8eac011940 comp rx=0 tx=0).stop 2024-09-16T01:20:49.632 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 shutdown_connections 2024-09-16T01:20:49.632 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57c7f8c0 0x560b57c81cf0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.632 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c6d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c6d250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 --2- 172.21.15.78:0/1722264612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.629+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 >> 172.21.15.78:0/1722264612 conn(0x560b57c50060 msgr2=0x560b57c55dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.633 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.630+0000 7f8eb37fe640 0 client.25921 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:49.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.635+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 shutdown_connections 2024-09-16T01:20:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.637+0000 7f8ecd054740 1 -- 172.21.15.78:0/1722264612 wait complete. 2024-09-16T01:20:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.637+0000 7f8ecd054740 20 client.25921 trim_cache size 0 max 16384 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.664+0000 7f8ecd054740 1 Processor -- start 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.664+0000 7f8ecd054740 1 -- start start 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c87b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a7060 con 0x560b57c539b0 2024-09-16T01:20:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32974/0 (socket says 172.21.15.78:32974) 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 -- 172.21.15.78:0/3267894498 learned_addr learned my addr 172.21.15.78:0/3267894498 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec885d640 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c87b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a4930 con 0x560b57c6fcc0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41974f0 con 0x560b57c542d0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 -- 172.21.15.78:0/3267894498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 msgr2=0x560b57c875d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c875d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 -- 172.21.15.78:0/3267894498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 msgr2=0x560b57c87b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c87b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.665+0000 7f8ec3fff640 1 -- 172.21.15.78:0/3267894498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b57c6bca0 con 0x560b57c539b0 2024-09-16T01:20:49.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.666+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40bc240 tx=0x7f8eb400a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6191bc0d475caf15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.666+0000 7f8eb9ffb640 1 -- 172.21.15.78:0/3267894498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40d4070 con 0x560b57c539b0 2024-09-16T01:20:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.666+0000 7f8eb9ffb640 1 -- 172.21.15.78:0/3267894498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb40dc040 con 0x560b57c539b0 2024-09-16T01:20:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.666+0000 7f8eb9ffb640 1 -- 172.21.15.78:0/3267894498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40f6040 con 0x560b57c539b0 2024-09-16T01:20:49.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 msgr2=0x560b57c6d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40bc240 tx=0x7f8eb400a370 comp rx=0 tx=0).stop 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 shutdown_connections 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c6fcc0 0x560b57c87b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c875d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3267894498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c539b0 0x560b57c6d3e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 >> 172.21.15.78:0/3267894498 conn(0x560b57c50060 msgr2=0x560b57c6ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 shutdown_connections 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- 172.21.15.78:0/3267894498 wait complete. 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 -- start start 2024-09-16T01:20:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.667+0000 7f8ecd054740 1 Processor -- start 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 0x560b57c737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c73d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 0x560b57c74250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a2e10 con 0x560b57c6fcc0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a4260 con 0x560b57c539b0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a34c0 con 0x560b57c542d0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ecd054740 10 client.0 _ll_register_callbacks cb 0x7f8eac0095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 0x560b57c737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 0x560b57c737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:39606/0 (socket says 172.21.15.78:39606) 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ec3fff640 1 -- 172.21.15.78:0/3804115055 learned_addr learned my addr 172.21.15.78:0/3804115055 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ec885d640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 0x560b57c74250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c73d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ebbfff640 1 -- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 msgr2=0x560b57c737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 0x560b57c737d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ebbfff640 1 -- 172.21.15.78:0/3804115055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 msgr2=0x560b57c74250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.668+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 0x560b57c74250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.669+0000 7f8ebbfff640 1 -- 172.21.15.78:0/3804115055 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b57c7c3c0 con 0x560b57c542d0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.669+0000 7f8ec885d640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 0x560b57c74250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8ebbfff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c73d10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x560b57c6e2c0 tx=0x7f8eac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e6785e8a3750c52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8eb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac0049a0 con 0x560b57c542d0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7f8eac004b40 con 0x560b57c542d0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb40dc040 con 0x560b57c542d0 2024-09-16T01:20:49.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8eac00f580 con 0x560b57c542d0 2024-09-16T01:20:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8ecd054740 10 client.25927 fetch_fsmap learned FSMap version 31 2024-09-16T01:20:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.670+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560b57cea820 con 0x560b57c542d0 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac004d40 con 0x560b57c542d0 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8eac0052d0 con 0x560b57c542d0 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 10 client.25927 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 10 client.25927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 20 client.25927 populate_metadata read hostname 'smithi078' 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57ceaef0 0x560b57ced310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 4 client.25927 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.671+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8eb40dc040 con 0x560b57ceaef0 2024-09-16T01:20:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.672+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57ceaef0 0x560b57ced310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:20:49.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.672+0000 7f8eb37fe640 10 client.25927 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:49.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.672+0000 7f8ec3fff640 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57ceaef0 0x560b57ced310 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba39d8df6ec78e server_cookie=c7165ab77ce57c3f in_seq=0 out_seq=0 2024-09-16T01:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8eb37fe640 1 -- 172.21.15.78:0/3804115055 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8eb40dc040 con 0x560b57ceaef0 2024-09-16T01:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8eb37fe640 10 client.25927 handle_command_reply: tid=0 2024-09-16T01:20:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 client.25927 shutdown 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 msgr2=0x560b57c73d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c73d10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x560b57c6e2c0 tx=0x7f8eac004730 comp rx=0 tx=0).stop 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 shutdown_connections 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560b57ceaef0 0x560b57ced310 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560b57c6fcc0 0x560b57c74250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560b57c542d0 0x560b57c73d10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 --2- 172.21.15.78:0/3804115055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560b57c539b0 0x560b57c737d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 >> 172.21.15.78:0/3804115055 conn(0x560b57c50060 msgr2=0x560b57c4f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8eb37fe640 0 client.25927 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 shutdown_connections 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 1 -- 172.21.15.78:0/3804115055 wait complete. 2024-09-16T01:20:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.680+0000 7f8ecd054740 20 client.25927 trim_cache size 0 max 16384 2024-09-16T01:20:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:20:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:20:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:20:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 msgr2=0x7f8ea40a9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 0x7f8ea40a9f20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc014bb0 tx=0x7f8ebc004ab0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 msgr2=0x7f8ec41961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402bc80 tx=0x7f8eb4030bf0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 shutdown_connections 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8ea40a7a60 0x7f8ea40a9f20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ec41026f0 0x7f8ec419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ec4101d70 0x7f8ec41966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 --2- 172.21.15.78:0/1772463706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec4100490 0x7f8ec41961a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:20:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 >> 172.21.15.78:0/1772463706 conn(0x7f8ec40770f0 msgr2=0x7f8ec4104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:20:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 shutdown_connections 2024-09-16T01:20:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:20:49.681+0000 7f8eca2c0640 1 -- 172.21.15.78:0/1772463706 wait complete. 2024-09-16T01:20:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 575.0 s, 834.4 tps, lat 117.706 ms stddev 140.483 2024-09-16T01:20:51.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:50 smithi078 ceph-mon[30937]: pgmap v312: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 553 op/s 2024-09-16T01:20:51.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:50 smithi193 ceph-mon[35701]: pgmap v312: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 553 op/s 2024-09-16T01:20:51.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:50 smithi148 ceph-mon[35030]: pgmap v312: 129 pgs: 129 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 553 op/s 2024-09-16T01:20:53.030 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:52 smithi078 ceph-mon[30937]: pgmap v313: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-09-16T01:20:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:52 smithi193 ceph-mon[35701]: pgmap v313: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-09-16T01:20:53.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:52 smithi148 ceph-mon[35030]: pgmap v313: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 593 op/s 2024-09-16T01:20:55.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:54 smithi078 ceph-mon[30937]: pgmap v314: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 460 op/s 2024-09-16T01:20:55.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:54 smithi193 ceph-mon[35701]: pgmap v314: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 460 op/s 2024-09-16T01:20:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:54 smithi148 ceph-mon[35030]: pgmap v314: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 460 op/s 2024-09-16T01:20:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 580.0 s, 835.4 tps, lat 108.442 ms stddev 138.533 2024-09-16T01:20:57.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:56 smithi078 ceph-mon[30937]: pgmap v315: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.8 MiB/s wr, 647 op/s 2024-09-16T01:20:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:56 smithi193 ceph-mon[35701]: pgmap v315: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.8 MiB/s wr, 647 op/s 2024-09-16T01:20:57.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:56 smithi148 ceph-mon[35030]: pgmap v315: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.8 MiB/s wr, 647 op/s 2024-09-16T01:20:58.940 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:20:58 smithi148 ceph-mon[35030]: pgmap v316: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.9 MiB/s wr, 664 op/s 2024-09-16T01:20:59.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:20:58 smithi078 ceph-mon[30937]: pgmap v316: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.9 MiB/s wr, 664 op/s 2024-09-16T01:20:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:20:58 smithi193 ceph-mon[35701]: pgmap v316: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.9 MiB/s wr, 664 op/s 2024-09-16T01:21:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 585.0 s, 860.2 tps, lat 127.139 ms stddev 175.110 2024-09-16T01:21:01.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:00 smithi078 ceph-mon[30937]: pgmap v317: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 898 op/s 2024-09-16T01:21:01.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:00 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:21:01.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:00 smithi193 ceph-mon[35701]: pgmap v317: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 898 op/s 2024-09-16T01:21:01.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:00 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:21:01.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:00 smithi148 ceph-mon[35030]: pgmap v317: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 898 op/s 2024-09-16T01:21:01.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:00 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:21:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 -- 172.21.15.78:0/1190561055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c101180 msgr2=0x7fc36c10d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 --2- 172.21.15.78:0/1190561055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c101180 0x7fc36c10d6b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc364009cd0 tx=0x7fc36402a8f0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 -- 172.21.15.78:0/1190561055 shutdown_connections 2024-09-16T01:21:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 --2- 172.21.15.78:0/1190561055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c101180 0x7fc36c10d6b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 --2- 172.21.15.78:0/1190561055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1007e0 0x7fc36c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 --2- 172.21.15.78:0/1190561055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c1067f0 0x7fc36c106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 -- 172.21.15.78:0/1190561055 >> 172.21.15.78:0/1190561055 conn(0x7fc36c0fc4e0 msgr2=0x7fc36c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 -- 172.21.15.78:0/1190561055 shutdown_connections 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.842+0000 7fc36b59e640 1 -- 172.21.15.78:0/1190561055 wait complete. 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 Processor -- start 2024-09-16T01:21:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 -- start start 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 0x7fc36c10dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 0x7fc36c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1067f0 0x7fc36c10ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c111d70 con 0x7fc36c1007e0 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c111bf0 con 0x7fc36c1067f0 2024-09-16T01:21:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.843+0000 7fc36b59e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c111ef0 con 0x7fc36c101180 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc36a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 0x7fc36c10dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc36ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1067f0 0x7fc36c10ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc36a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 0x7fc36c10dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34054/0 (socket says 172.21.15.78:34054) 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc36a59c640 1 -- 172.21.15.78:0/2099507078 learned_addr learned my addr 172.21.15.78:0/2099507078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 0x7fc36c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 -- 172.21.15.78:0/2099507078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1067f0 msgr2=0x7fc36c10ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1067f0 0x7fc36c10ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 -- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 msgr2=0x7fc36c10dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 0x7fc36c10dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 -- 172.21.15.78:0/2099507078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c1a6720 con 0x7fc36c101180 2024-09-16T01:21:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc369d9b640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 0x7fc36c10e500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc35800e9f0 tx=0x7fc35800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.844+0000 7fc3637fe640 1 -- 172.21.15.78:0/2099507078 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35800cd80 con 0x7fc36c101180 2024-09-16T01:21:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.845+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c1a6a10 con 0x7fc36c101180 2024-09-16T01:21:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.845+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c1a6fb0 con 0x7fc36c101180 2024-09-16T01:21:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.845+0000 7fc3637fe640 1 -- 172.21.15.78:0/2099507078 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3580035f0 con 0x7fc36c101180 2024-09-16T01:21:02.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.845+0000 7fc3637fe640 1 -- 172.21.15.78:0/2099507078 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358010690 con 0x7fc36c101180 2024-09-16T01:21:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.848+0000 7fc3637fe640 1 -- 172.21.15.78:0/2099507078 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fc358003fa0 con 0x7fc36c101180 2024-09-16T01:21:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.848+0000 7fc3739ff740 1 Processor -- start 2024-09-16T01:21:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 -- start start 2024-09-16T01:21:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 0x559de84f5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559de84462f0 con 0x559de84ea4d0 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559de83ee270 con 0x559de84eadf0 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc3739ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559de84422b0 con 0x559de84f5690 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.849+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34414/0 (socket says 172.21.15.78:34414) 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 -- 172.21.15.78:0/3297192308 learned_addr learned my addr 172.21.15.78:0/3297192308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc36a59c640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 0x559de84f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 -- 172.21.15.78:0/3297192308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 msgr2=0x559de84f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 0x559de84f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 -- 172.21.15.78:0/3297192308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 msgr2=0x559de84ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.850+0000 7fc369d9b640 1 -- 172.21.15.78:0/3297192308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de83ed340 con 0x559de84f5690 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.851+0000 7fc36a59c640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 0x559de84f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.851+0000 7fc369d9b640 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3580cd4f0 tx=0x7fc3580d86f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae65f91897f6164 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.852+0000 7fc360ff9640 1 -- 172.21.15.78:0/3297192308 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580e3070 con 0x559de84f5690 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.852+0000 7fc360ff9640 1 -- 172.21.15.78:0/3297192308 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3580d0b40 con 0x559de84f5690 2024-09-16T01:21:02.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.852+0000 7fc360ff9640 1 -- 172.21.15.78:0/3297192308 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580deac0 con 0x559de84f5690 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.852+0000 7fc3637fe640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 0x7fc3500a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.853+0000 7fc3637fe640 1 -- 172.21.15.78:0/2099507078 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fc3580d02b0 con 0x7fc36c101180 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.853+0000 7fc3739ff740 1 -- 172.21.15.78:0/3297192308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 msgr2=0x559de84f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.853+0000 7fc3739ff740 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3580cd4f0 tx=0x7fc3580d86f0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 0x7fc3500a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc3739ff740 1 -- 172.21.15.78:0/3297192308 shutdown_connections 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc3739ff740 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84f7a80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc3739ff740 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84eadf0 0x559de84f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc3739ff740 1 --2- 172.21.15.78:0/3297192308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.854+0000 7fc3739ff740 1 -- 172.21.15.78:0/3297192308 >> 172.21.15.78:0/3297192308 conn(0x559de84e6a40 msgr2=0x559de84e6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.855+0000 7fc3739ff740 1 -- 172.21.15.78:0/3297192308 shutdown_connections 2024-09-16T01:21:02.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.855+0000 7fc3739ff740 1 -- 172.21.15.78:0/3297192308 wait complete. 2024-09-16T01:21:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.856+0000 7fc3739ff740 1 -- start start 2024-09-16T01:21:02.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.856+0000 7fc3739ff740 1 Processor -- start 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.856+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.856+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.856+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de84f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc3739ff740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x559de84f8260 con 0x559de84ea4d0 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc3739ff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559de83cf290 con 0x559de84f5690 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc3739ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559de84f83e0 con 0x559de84eadf0 2024-09-16T01:21:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc3739ff740 10 client.0 _ll_register_callbacks cb 0x559de8400140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 0x7fc3500a5970 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc364004af0 tx=0x7fc364005d80 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34420/0 (socket says 172.21.15.78:34420) 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc369d9b640 1 -- 172.21.15.78:0/1389101357 learned_addr learned my addr 172.21.15.78:0/1389101357 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.857+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 -- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 msgr2=0x559de84f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de84f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 -- 172.21.15.78:0/1389101357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 msgr2=0x559de84f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 -- 172.21.15.78:0/1389101357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de8159210 con 0x559de84eadf0 2024-09-16T01:21:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.858+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc35809c0d0 tx=0x7fc3580d2590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3aef3e1175eba54f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.859+0000 7fc34dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:21:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.861+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.866+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3580983e0 con 0x559de84eadf0 2024-09-16T01:21:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.866+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3580d0b40 con 0x559de84eadf0 2024-09-16T01:21:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.866+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fc35809a440 con 0x559de84eadf0 2024-09-16T01:21:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.875+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc35809baa0 con 0x559de84eadf0 2024-09-16T01:21:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.879+0000 7fc3739ff740 10 client.25939 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:02.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.879+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559de8225620 con 0x559de84eadf0 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3580d0b40 con 0x559de84eadf0 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc3580987b0 con 0x559de84eadf0 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 10 client.25939 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 10 client.25939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 20 client.25939 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de8505610 0x559de8507a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 4 client.25939 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.885+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559de8453a20 con 0x559de8505610 2024-09-16T01:21:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.887+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de8505610 0x559de8507a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.900+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de8505610 0x559de8507a00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e1464b2016e8ebc server_cookie=ee5c8761c009cd51 in_seq=0 out_seq=0 2024-09-16T01:21:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.900+0000 7fc34dffb640 10 client.25939 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:02.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.905+0000 7fc34dffb640 1 -- 172.21.15.78:0/1389101357 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559de8453a20 con 0x559de8505610 2024-09-16T01:21:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.905+0000 7fc34dffb640 10 client.25939 handle_command_reply: tid=0 2024-09-16T01:21:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.905+0000 7fc3739ff740 1 client.25939 shutdown 2024-09-16T01:21:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.915+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 msgr2=0x559de84f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.915+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc35809c0d0 tx=0x7fc3580d2590 comp rx=0 tx=0).stop 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 shutdown_connections 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de8505610 0x559de8507a00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de84f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84f2ac0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1389101357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.924+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 >> 172.21.15.78:0/1389101357 conn(0x559de84e6a40 msgr2=0x559de84ec7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.926+0000 7fc34dffb640 0 client.25939 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.927+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 shutdown_connections 2024-09-16T01:21:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.928+0000 7fc3739ff740 1 -- 172.21.15.78:0/1389101357 wait complete. 2024-09-16T01:21:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.930+0000 7fc3739ff740 20 client.25939 trim_cache size 0 max 16384 2024-09-16T01:21:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.959+0000 7fc3739ff740 1 Processor -- start 2024-09-16T01:21:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.959+0000 7fc3739ff740 1 -- start start 2024-09-16T01:21:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.959+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.959+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84dd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.959+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de850d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc3739ff740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc354012950 con 0x559de84ea4d0 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc3739ff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc354010220 con 0x559de84f5690 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc3739ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc35400e090 con 0x559de84eadf0 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84dd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84dd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:34424/0 (socket says 172.21.15.78:34424) 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc369d9b640 1 -- 172.21.15.78:0/1850034947 learned_addr learned my addr 172.21.15.78:0/1850034947 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.960+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de850d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 -- 172.21.15.78:0/1850034947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 msgr2=0x559de84dd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84dd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 -- 172.21.15.78:0/1850034947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 msgr2=0x559de84f2e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 -- 172.21.15.78:0/1850034947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de84ea250 con 0x559de84f5690 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de850d310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc36402ae10 tx=0x7fc3640095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9301e624f753d808 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc34ffff640 1 -- 172.21.15.78:0/1850034947 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364003bf0 con 0x559de84f5690 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc34ffff640 1 -- 172.21.15.78:0/1850034947 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc364036050 con 0x559de84f5690 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.961+0000 7fc34ffff640 1 -- 172.21.15.78:0/1850034947 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364004640 con 0x559de84f5690 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- 172.21.15.78:0/1850034947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 msgr2=0x559de850d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de850d310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc36402ae10 tx=0x7fc3640095c0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- 172.21.15.78:0/1850034947 shutdown_connections 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84f5690 0x559de850d310 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84eadf0 0x559de84dd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1850034947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84ea4d0 0x559de84f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- 172.21.15.78:0/1850034947 >> 172.21.15.78:0/1850034947 conn(0x559de84e6a40 msgr2=0x559de84dca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- 172.21.15.78:0/1850034947 shutdown_connections 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- 172.21.15.78:0/1850034947 wait complete. 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.962+0000 7fc3739ff740 1 -- start start 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 Processor -- start 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84ea4d0 0x559de8568a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc35400e6e0 con 0x559de84eadf0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc35400fb50 con 0x559de84ea4d0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc35400edb0 con 0x559de84f5690 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc3739ff740 10 client.0 _ll_register_callbacks cb 0x7fc3580e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc36a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84ea4d0 0x559de8568a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34088/0 (socket says 172.21.15.78:34088) 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 -- 172.21.15.78:0/1431027894 learned_addr learned my addr 172.21.15.78:0/1431027894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 -- 172.21.15.78:0/1431027894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 msgr2=0x559de84fc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84fc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 -- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84ea4d0 msgr2=0x559de8568a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84ea4d0 0x559de8568a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc369d9b640 1 -- 172.21.15.78:0/1431027894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de84fcc30 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.963+0000 7fc36ad9d640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc369d9b640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc35809c0d0 tx=0x7fc3580d8f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239ad9fd88506399 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35809bc80 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9525+0+0 (secure 0 0 0) 0x7fc3580987b0 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc364036050 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc35809bf10 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 10 client.16554 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559de84fdf10 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc364036050 con 0x559de84eadf0 2024-09-16T01:21:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc3580a0150 con 0x559de84eadf0 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 10 client.16554 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 20 client.16554 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de84e1ba0 0x559de84e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 4 client.16554 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.964+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559de844dd70 con 0x559de84e1ba0 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.965+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de84e1ba0 0x559de84e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.965+0000 7fc34dffb640 10 client.16554 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.965+0000 7fc36a59c640 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de84e1ba0 0x559de84e3fc0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df95bccc40e8ae72 server_cookie=96246a0308c8936b in_seq=0 out_seq=0 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.973+0000 7fc34dffb640 1 -- 172.21.15.78:0/1431027894 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559de844dd70 con 0x559de84e1ba0 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.973+0000 7fc34dffb640 10 client.16554 handle_command_reply: tid=0 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.973+0000 7fc3739ff740 1 client.16554 shutdown 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 msgr2=0x559de8568f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc35809c0d0 tx=0x7fc3580d8f20 comp rx=0 tx=0).stop 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 shutdown_connections 2024-09-16T01:21:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x559de84e1ba0 0x559de84e3fc0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de84f5690 0x559de84fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559de84eadf0 0x559de8568f90 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 --2- 172.21.15.78:0/1431027894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559de84ea4d0 0x559de8568a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 >> 172.21.15.78:0/1431027894 conn(0x559de84e6a40 msgr2=0x559de84ec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc34dffb640 0 client.16554 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 shutdown_connections 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 1 -- 172.21.15.78:0/1431027894 wait complete. 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc3739ff740 20 client.16554 trim_cache size 0 max 16384 2024-09-16T01:21:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:21:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:21:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:21:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:21:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 msgr2=0x7fc3500a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 0x7fc3500a5970 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc364004af0 tx=0x7fc364005d80 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 msgr2=0x7fc36c10e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.974+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 0x7fc36c10e500 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc35800e9f0 tx=0x7fc35800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 shutdown_connections 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc3500a34b0 0x7fc3500a5970 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc36c1067f0 0x7fc36c10ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc36c101180 0x7fc36c10e500 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 --2- 172.21.15.78:0/2099507078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c1007e0 0x7fc36c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 >> 172.21.15.78:0/2099507078 conn(0x7fc36c0fc4e0 msgr2=0x7fc36c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 shutdown_connections 2024-09-16T01:21:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:02.975+0000 7fc36b59e640 1 -- 172.21.15.78:0/2099507078 wait complete. 2024-09-16T01:21:03.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:02 smithi078 ceph-mon[30937]: pgmap v318: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.2 MiB/s wr, 1.02k op/s 2024-09-16T01:21:03.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:02 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:03.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:02 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:02 smithi193 ceph-mon[35701]: pgmap v318: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.2 MiB/s wr, 1.02k op/s 2024-09-16T01:21:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:02 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:02 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:03.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:02 smithi148 ceph-mon[35030]: pgmap v318: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.2 MiB/s wr, 1.02k op/s 2024-09-16T01:21:03.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:02 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:03.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:02 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:04.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:21:04.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:03 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:21:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:21:04.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:03 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:21:04.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:04.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-16T01:21:04.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:03 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi148 to 184589619: error parsing value: Value '184589619' is below minimum 939524096 2024-09-16T01:21:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: pgmap v319: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.5 MiB/s wr, 968 op/s 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:05 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: pgmap v319: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.5 MiB/s wr, 968 op/s 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:05 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: pgmap v319: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.5 MiB/s wr, 968 op/s 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:05 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 590.0 s, 632.8 tps, lat 160.346 ms stddev 208.070 2024-09-16T01:21:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:06 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:21:06.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:06 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:21:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:06 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:21:06.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:06 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:21:06.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:06 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi078 to 176.0M 2024-09-16T01:21:06.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:06 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi078 to 184555929: error parsing value: Value '184555929' is below minimum 939524096 2024-09-16T01:21:07.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:07 smithi148 ceph-mon[35030]: pgmap v320: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-09-16T01:21:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:07 smithi078 ceph-mon[30937]: pgmap v320: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-09-16T01:21:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:07 smithi193 ceph-mon[35701]: pgmap v320: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: pgmap v321: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:21:09.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:21:09.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: pgmap v321: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:21:09.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:21:09.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: pgmap v321: 129 pgs: 129 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-09-16T01:21:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: osdmap e103: 12 total, 12 up, 12 in 2024-09-16T01:21:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:21:09.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:21:10.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:10 smithi148 ceph-mon[35030]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:21:10.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:10 smithi148 ceph-mon[35030]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:21:10.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:10 smithi078 ceph-mon[30937]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:21:10.439 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:10 smithi078 ceph-mon[30937]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:21:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:10 smithi193 ceph-mon[35701]: Adjusting osd_memory_target on smithi193 to 176.0M 2024-09-16T01:21:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:10 smithi193 ceph-mon[35701]: Unable to set osd_memory_target on smithi193 to 184595353: error parsing value: Value '184595353' is below minimum 939524096 2024-09-16T01:21:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 595.0 s, 581.2 tps, lat 168.580 ms stddev 220.661 2024-09-16T01:21:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:11 smithi148 ceph-mon[35030]: pgmap v323: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 775 op/s 2024-09-16T01:21:11.453 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:11 smithi078 ceph-mon[30937]: pgmap v323: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 775 op/s 2024-09-16T01:21:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:11 smithi193 ceph-mon[35701]: pgmap v323: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 775 op/s 2024-09-16T01:21:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:13 smithi148 ceph-mon[35030]: pgmap v324: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.5 MiB/s wr, 608 op/s 2024-09-16T01:21:13.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:13 smithi078 ceph-mon[30937]: pgmap v324: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.5 MiB/s wr, 608 op/s 2024-09-16T01:21:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:13 smithi193 ceph-mon[35701]: pgmap v324: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.5 MiB/s wr, 608 op/s 2024-09-16T01:21:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:15 smithi148 ceph-mon[35030]: pgmap v325: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.3 MiB/s wr, 563 op/s 2024-09-16T01:21:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:15 smithi078 ceph-mon[30937]: pgmap v325: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.3 MiB/s wr, 563 op/s 2024-09-16T01:21:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:15 smithi193 ceph-mon[35701]: pgmap v325: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 5.3 MiB/s wr, 563 op/s 2024-09-16T01:21:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 600.0 s, 577.4 tps, lat 177.299 ms stddev 228.557 2024-09-16T01:21:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 -- 172.21.15.78:0/2423708160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6841009b0 msgr2=0x7fa684100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 --2- 172.21.15.78:0/2423708160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6841009b0 0x7fa684100e10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa678009930 tx=0x7fa67802a420 comp rx=0 tx=0).stop 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 -- 172.21.15.78:0/2423708160 shutdown_connections 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 --2- 172.21.15.78:0/2423708160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa684101350 0x7fa68410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 --2- 172.21.15.78:0/2423708160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6841009b0 0x7fa684100e10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 --2- 172.21.15.78:0/2423708160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841069c0 0x7fa684106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 -- 172.21.15.78:0/2423708160 >> 172.21.15.78:0/2423708160 conn(0x7fa684077990 msgr2=0x7fa6840fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 -- 172.21.15.78:0/2423708160 shutdown_connections 2024-09-16T01:21:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.103+0000 7fa68892c640 1 -- 172.21.15.78:0/2423708160 wait complete. 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 Processor -- start 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 -- start start 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841009b0 0x7fa68410d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 0x7fa68410b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410f2f0 con 0x7fa684101350 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410f170 con 0x7fa6841009b0 2024-09-16T01:21:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa68892c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410f470 con 0x7fa6841069c0 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.104+0000 7fa681d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37098/0 (socket says 172.21.15.78:37098) 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 -- 172.21.15.78:0/210947566 learned_addr learned my addr 172.21.15.78:0/210947566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa682d9d640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 0x7fa68410b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa68259c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841009b0 0x7fa68410d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 -- 172.21.15.78:0/210947566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 msgr2=0x7fa68410b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 0x7fa68410b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 -- 172.21.15.78:0/210947566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841009b0 msgr2=0x7fa68410d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841009b0 0x7fa68410d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 -- 172.21.15.78:0/210947566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68410b9f0 con 0x7fa684101350 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa682d9d640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 0x7fa68410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.105+0000 7fa681d9b640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa678009230 tx=0x7fa678004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.106+0000 7fa6737fe640 1 -- 172.21.15.78:0/210947566 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67803f030 con 0x7fa684101350 2024-09-16T01:21:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.106+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68410bc80 con 0x7fa684101350 2024-09-16T01:21:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.106+0000 7fa6737fe640 1 -- 172.21.15.78:0/210947566 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa678004440 con 0x7fa684101350 2024-09-16T01:21:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.107+0000 7fa6737fe640 1 -- 172.21.15.78:0/210947566 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678049060 con 0x7fa684101350 2024-09-16T01:21:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.107+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68410c0b0 con 0x7fa684101350 2024-09-16T01:21:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.110+0000 7fa6737fe640 1 -- 172.21.15.78:0/210947566 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fa678042030 con 0x7fa684101350 2024-09-16T01:21:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.111+0000 7fa6737fe640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 0x7fa66c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.111+0000 7fa6737fe640 1 -- 172.21.15.78:0/210947566 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9613+0+0 (secure 0 0 0) 0x7fa6780efd60 con 0x7fa684101350 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.111+0000 7fa68259c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 0x7fa66c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.111+0000 7fa68b6c2740 1 Processor -- start 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.111+0000 7fa68b6c2740 1 -- start start 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 0x7fa66c0a5920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa674005fd0 tx=0x7fa674009500 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffcfa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffcfd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5616ffc542f0 con 0x5616ffcefaf0 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616ffbfc270 con 0x5616ffcfacb0 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5616ffc502b0 con 0x5616ffcf0410 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37104/0 (socket says 172.21.15.78:37104) 2024-09-16T01:21:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 -- 172.21.15.78:0/1863395520 learned_addr learned my addr 172.21.15.78:0/1863395520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa681d9b640 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffcfa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa682d9d640 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffcfd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 -- 172.21.15.78:0/1863395520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 msgr2=0x5616ffcfa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffcfa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 -- 172.21.15.78:0/1863395520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 msgr2=0x5616ffcfd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffcfd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 -- 172.21.15.78:0/1863395520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ffbfb340 con 0x5616ffcefaf0 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.112+0000 7fa68259c640 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa674003bf0 tx=0x7fa674014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56203ad563a4405 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.113+0000 7fa670ff9640 1 -- 172.21.15.78:0/1863395520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674026430 con 0x5616ffcefaf0 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.113+0000 7fa670ff9640 1 -- 172.21.15.78:0/1863395520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa67401c900 con 0x5616ffcefaf0 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.113+0000 7fa670ff9640 1 -- 172.21.15.78:0/1863395520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67401da40 con 0x5616ffcefaf0 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 -- 172.21.15.78:0/1863395520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 msgr2=0x5616ffcefed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa674003bf0 tx=0x7fa674014000 comp rx=0 tx=0).stop 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 -- 172.21.15.78:0/1863395520 shutdown_connections 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffcfd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffcfa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/1863395520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcefed0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.114+0000 7fa68b6c2740 1 -- 172.21.15.78:0/1863395520 >> 172.21.15.78:0/1863395520 conn(0x5616ffcec060 msgr2=0x5616ffcec440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.115+0000 7fa68b6c2740 1 -- 172.21.15.78:0/1863395520 shutdown_connections 2024-09-16T01:21:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.116+0000 7fa68b6c2740 1 -- 172.21.15.78:0/1863395520 wait complete. 2024-09-16T01:21:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.116+0000 7fa68b6c2740 1 -- start start 2024-09-16T01:21:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.116+0000 7fa68b6c2740 1 Processor -- start 2024-09-16T01:21:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffcf85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5616ffcfd880 con 0x5616ffcfacb0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616ffbdd290 con 0x5616ffcf0410 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa681d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68259c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa681d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45656/0 (socket says 172.21.15.78:45656) 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa681d9b640 1 -- 172.21.15.78:0/2002102560 learned_addr learned my addr 172.21.15.78:0/2002102560 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa682d9d640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffcf85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.117+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5616ffcfda00 con 0x5616ffcefaf0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa68b6c2740 10 client.0 _ll_register_callbacks cb 0x5616ffc0e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 -- 172.21.15.78:0/2002102560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 msgr2=0x5616ffcf7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 -- 172.21.15.78:0/2002102560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 msgr2=0x5616ffcf85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffcf85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 -- 172.21.15.78:0/2002102560 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ff967210 con 0x5616ffcf0410 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa682d9d640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffcf85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6780096c0 tx=0x7fa6780bb6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59fb54fb9f62736 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678035050 con 0x5616ffcf0410 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9613+0+0 (secure 0 0 0) 0x7fa678031bd0 con 0x5616ffcf0410 2024-09-16T01:21:16.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.118+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa67401c900 con 0x5616ffcf0410 2024-09-16T01:21:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.122+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa678031dd0 con 0x5616ffcf0410 2024-09-16T01:21:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.122+0000 7fa68b6c2740 10 client.25904 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.122+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5616ffa33620 con 0x5616ffcf0410 2024-09-16T01:21:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.123+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa67401c900 con 0x5616ffcf0410 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.123+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa6780bb100 con 0x5616ffcf0410 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 10 client.25904 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 10 client.25904 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 20 client.25904 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd0abd0 0x5616ffd0d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 4 client.25904 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616ffc61a20 con 0x5616ffd0abd0 2024-09-16T01:21:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68259c640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd0abd0 0x5616ffd0d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa68259c640 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd0abd0 0x5616ffd0d000 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95d672dd8cc6f1c4 server_cookie=1b84a651557425f0 in_seq=0 out_seq=0 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.124+0000 7fa669ffb640 10 client.25904 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa669ffb640 1 -- 172.21.15.78:0/2002102560 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616ffc61a20 con 0x5616ffd0abd0 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa669ffb640 10 client.25904 handle_command_reply: tid=0 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 client.25904 shutdown 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 msgr2=0x5616ffcf8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6780096c0 tx=0x7fa6780bb6d0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 shutdown_connections 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd0abd0 0x5616ffd0d000 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffcf85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf8060 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2002102560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 >> 172.21.15.78:0/2002102560 conn(0x5616ffcec060 msgr2=0x5616ffcf1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa669ffb640 0 client.25904 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 shutdown_connections 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.125+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2002102560 wait complete. 2024-09-16T01:21:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.126+0000 7fa68b6c2740 20 client.25904 trim_cache size 0 max 16384 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.150+0000 7fa68b6c2740 1 Processor -- start 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 -- start start 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffd12910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffd12e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b8000 con 0x5616ffcefaf0 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b58d0 con 0x5616ffcfacb0 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b3740 con 0x5616ffcf0410 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.157 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37122/0 (socket says 172.21.15.78:37122) 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.151+0000 7fa68259c640 1 -- 172.21.15.78:0/3397793762 learned_addr learned my addr 172.21.15.78:0/3397793762 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa682d9d640 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffd12e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 -- 172.21.15.78:0/3397793762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 msgr2=0x5616ffd12910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffd12910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 -- 172.21.15.78:0/3397793762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 msgr2=0x5616ffd12e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffd12e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 -- 172.21.15.78:0/3397793762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ffac38a0 con 0x5616ffcefaf0 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa682d9d640 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffd12e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa68259c640 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa674009a00 tx=0x7fa674009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cd16c26bf1e7f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.152+0000 7fa66bfff640 1 -- 172.21.15.78:0/3397793762 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674004740 con 0x5616ffcefaf0 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.153+0000 7fa66bfff640 1 -- 172.21.15.78:0/3397793762 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa674004dc0 con 0x5616ffcefaf0 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.153+0000 7fa68b6c2740 1 -- 172.21.15.78:0/3397793762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 msgr2=0x5616ffcf83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.153+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa674009a00 tx=0x7fa674009990 comp rx=0 tx=0).stop 2024-09-16T01:21:16.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.158+0000 7fa68b6c2740 1 -- 172.21.15.78:0/3397793762 shutdown_connections 2024-09-16T01:21:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.158+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcfacb0 0x5616ffd12e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.158+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcf0410 0x5616ffd12910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.158+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/3397793762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcefaf0 0x5616ffcf83d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.158+0000 7fa68b6c2740 1 -- 172.21.15.78:0/3397793762 >> 172.21.15.78:0/3397793762 conn(0x5616ffcec060 msgr2=0x5616ffcec440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.163+0000 7fa68b6c2740 1 -- 172.21.15.78:0/3397793762 shutdown_connections 2024-09-16T01:21:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.166+0000 7fa68b6c2740 1 -- 172.21.15.78:0/3397793762 wait complete. 2024-09-16T01:21:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.167+0000 7fa68b6c2740 1 -- start start 2024-09-16T01:21:16.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.167+0000 7fa68b6c2740 1 Processor -- start 2024-09-16T01:21:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffd01c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b3d90 con 0x5616ffcfacb0 2024-09-16T01:21:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b5200 con 0x5616ffcf0410 2024-09-16T01:21:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b4460 con 0x5616ffcefaf0 2024-09-16T01:21:16.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.171+0000 7fa68b6c2740 10 client.0 _ll_register_callbacks cb 0x7fa6780bb7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:16.174 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.172+0000 7fa681d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.177+0000 7fa682d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffd01c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.177+0000 7fa682d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffd01c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37138/0 (socket says 172.21.15.78:37138) 2024-09-16T01:21:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.177+0000 7fa682d9d640 1 -- 172.21.15.78:0/2573971521 learned_addr learned my addr 172.21.15.78:0/2573971521 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.177+0000 7fa68259c640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.181+0000 7fa681d9b640 1 -- 172.21.15.78:0/2573971521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 msgr2=0x5616ffcf4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.181+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.181+0000 7fa681d9b640 1 -- 172.21.15.78:0/2573971521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 msgr2=0x5616ffd01c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.181+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffd01c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.181+0000 7fa681d9b640 1 -- 172.21.15.78:0/2573971521 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ffd021d0 con 0x5616ffcf0410 2024-09-16T01:21:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.185+0000 7fa68259c640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:21:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.187+0000 7fa681d9b640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf4850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5616ffbf7550 tx=0x7fa6780c4610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7103d0b99f246f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.187+0000 7fa669ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:21:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.191+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678035070 con 0x5616ffcf0410 2024-09-16T01:21:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.191+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9613+0+0 (secure 0 0 0) 0x7fa678031bd0 con 0x5616ffcf0410 2024-09-16T01:21:16.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.192+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa674004dc0 con 0x5616ffcf0410 2024-09-16T01:21:16.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.197+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6780b7c10 con 0x5616ffcf0410 2024-09-16T01:21:16.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.198+0000 7fa68b6c2740 10 client.25907 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:16.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.198+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5616ffd03450 con 0x5616ffcf0410 2024-09-16T01:21:16.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.204+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa674004dc0 con 0x5616ffcf0410 2024-09-16T01:21:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa6780a1050 con 0x5616ffcf0410 2024-09-16T01:21:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 10 client.25907 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 10 client.25907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 20 client.25907 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd868f0 0x5616ffd88ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 4 client.25907 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:21:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.205+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616ffc5bd70 con 0x5616ffd868f0 2024-09-16T01:21:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.210+0000 7fa68259c640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd868f0 0x5616ffd88ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.218+0000 7fa68259c640 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd868f0 0x5616ffd88ce0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55659079a886075 server_cookie=ac990d0935ec1dd5 in_seq=0 out_seq=0 2024-09-16T01:21:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.219+0000 7fa669ffb640 10 client.25907 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa669ffb640 1 -- 172.21.15.78:0/2573971521 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616ffc5bd70 con 0x5616ffd868f0 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa669ffb640 10 client.25907 handle_command_reply: tid=0 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 client.25907 shutdown 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 msgr2=0x5616ffcf4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf4850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5616ffbf7550 tx=0x7fa6780c4610 comp rx=0 tx=0).stop 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 shutdown_connections 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5616ffd868f0 0x5616ffd88ce0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ffcfacb0 0x5616ffd01c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ffcf0410 0x5616ffcf4850 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 --2- 172.21.15.78:0/2573971521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616ffcefaf0 0x5616ffcf4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 >> 172.21.15.78:0/2573971521 conn(0x5616ffcec060 msgr2=0x5616ffcea9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa669ffb640 0 client.25907 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 shutdown_connections 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.316+0000 7fa68b6c2740 1 -- 172.21.15.78:0/2573971521 wait complete. 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.317+0000 7fa68b6c2740 20 client.25907 trim_cache size 0 max 16384 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:21:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 msgr2=0x7fa66c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 0x7fa66c0a5920 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa674005fd0 tx=0x7fa674009500 comp rx=0 tx=0).stop 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 msgr2=0x7fa68410d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa678009230 tx=0x7fa678004040 comp rx=0 tx=0).stop 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 shutdown_connections 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa66c0a3460 0x7fa66c0a5920 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6841069c0 0x7fa68410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa684101350 0x7fa68410d580 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 --2- 172.21.15.78:0/210947566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841009b0 0x7fa68410d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 >> 172.21.15.78:0/210947566 conn(0x7fa684077990 msgr2=0x7fa6841037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 shutdown_connections 2024-09-16T01:21:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:16.318+0000 7fa68892c640 1 -- 172.21.15.78:0/210947566 wait complete. 2024-09-16T01:21:17.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:17 smithi148 ceph-mon[35030]: pgmap v326: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 542 op/s 2024-09-16T01:21:17.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:17 smithi078 ceph-mon[30937]: pgmap v326: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 542 op/s 2024-09-16T01:21:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:17 smithi193 ceph-mon[35701]: pgmap v326: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.4 MiB/s wr, 542 op/s 2024-09-16T01:21:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:19 smithi148 ceph-mon[35030]: pgmap v327: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.9 MiB/s wr, 571 op/s 2024-09-16T01:21:19.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:19 smithi078 ceph-mon[30937]: pgmap v327: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.9 MiB/s wr, 571 op/s 2024-09-16T01:21:19.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:19 smithi193 ceph-mon[35701]: pgmap v327: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.9 MiB/s wr, 571 op/s 2024-09-16T01:21:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 605.0 s, 687.6 tps, lat 143.632 ms stddev 161.735 2024-09-16T01:21:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:21 smithi078 ceph-mon[30937]: pgmap v328: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.5 MiB/s wr, 624 op/s 2024-09-16T01:21:21.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:21 smithi078 ceph-mon[30937]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T01:21:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:21 smithi193 ceph-mon[35701]: pgmap v328: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.5 MiB/s wr, 624 op/s 2024-09-16T01:21:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:21 smithi193 ceph-mon[35701]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T01:21:21.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:21 smithi148 ceph-mon[35030]: pgmap v328: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.5 MiB/s wr, 624 op/s 2024-09-16T01:21:21.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:21 smithi148 ceph-mon[35030]: osdmap e104: 12 total, 12 up, 12 in 2024-09-16T01:21:23.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:23 smithi078 ceph-mon[30937]: pgmap v330: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.8 MiB/s wr, 610 op/s 2024-09-16T01:21:23.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:23 smithi193 ceph-mon[35701]: pgmap v330: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.8 MiB/s wr, 610 op/s 2024-09-16T01:21:23.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:23 smithi148 ceph-mon[35030]: pgmap v330: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 9.8 MiB/s wr, 610 op/s 2024-09-16T01:21:25.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:25 smithi078 ceph-mon[30937]: pgmap v331: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 10 MiB/s wr, 627 op/s 2024-09-16T01:21:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:25 smithi193 ceph-mon[35701]: pgmap v331: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 10 MiB/s wr, 627 op/s 2024-09-16T01:21:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:25 smithi148 ceph-mon[35030]: pgmap v331: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 10 MiB/s wr, 627 op/s 2024-09-16T01:21:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 610.0 s, 593.0 tps, lat 157.581 ms stddev 209.005 2024-09-16T01:21:27.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:27 smithi078 ceph-mon[30937]: pgmap v332: 129 pgs: 129 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 616 op/s 2024-09-16T01:21:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:27 smithi193 ceph-mon[35701]: pgmap v332: 129 pgs: 129 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 616 op/s 2024-09-16T01:21:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:27 smithi148 ceph-mon[35030]: pgmap v332: 129 pgs: 129 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 616 op/s 2024-09-16T01:21:29.437 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:29 smithi148 ceph-mon[35030]: pgmap v333: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 630 op/s 2024-09-16T01:21:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.473+0000 7f2c75531640 1 -- 172.21.15.78:0/4022052735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 msgr2=0x7f2c70109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.473+0000 7f2c75531640 1 --2- 172.21.15.78:0/4022052735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70109a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009cd0 tx=0x7f2c6802a930 comp rx=0 tx=0).stop 2024-09-16T01:21:29.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.477+0000 7f2c75531640 1 -- 172.21.15.78:0/4022052735 shutdown_connections 2024-09-16T01:21:29.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.477+0000 7f2c75531640 1 --2- 172.21.15.78:0/4022052735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7011b140 0x7f2c7011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.477+0000 7f2c75531640 1 --2- 172.21.15.78:0/4022052735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70109a10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.477+0000 7f2c75531640 1 --2- 172.21.15.78:0/4022052735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c701091f0 0x7f2c70110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.477+0000 7f2c75531640 1 -- 172.21.15.78:0/4022052735 >> 172.21.15.78:0/4022052735 conn(0x7f2c7006ce00 msgr2=0x7f2c7006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.485+0000 7f2c75531640 1 -- 172.21.15.78:0/4022052735 shutdown_connections 2024-09-16T01:21:29.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.488+0000 7f2c75531640 1 -- 172.21.15.78:0/4022052735 wait complete. 2024-09-16T01:21:29.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.488+0000 7f2c75531640 1 Processor -- start 2024-09-16T01:21:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.490+0000 7f2c75531640 1 -- start start 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.490+0000 7f2c75531640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.490+0000 7f2c75531640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70114970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.491+0000 7f2c75531640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c7011b140 0x7f2c70114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.491+0000 7f2c75531640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2c70111370 con 0x7f2c701091f0 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.491+0000 7f2c75531640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701111f0 con 0x7f2c701095d0 2024-09-16T01:21:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.491+0000 7f2c75531640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701114f0 con 0x7f2c7011b140 2024-09-16T01:21:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.495+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.495+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35304/0 (socket says 172.21.15.78:35304) 2024-09-16T01:21:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.495+0000 7f2c6effd640 1 -- 172.21.15.78:0/2965695894 learned_addr learned my addr 172.21.15.78:0/2965695894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:29.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.496+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70114970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.496+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c7011b140 0x7f2c70114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 -- 172.21.15.78:0/2965695894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c7011b140 msgr2=0x7f2c70114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c7011b140 0x7f2c70114eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 -- 172.21.15.78:0/2965695894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 msgr2=0x7f2c70114970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70114970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 -- 172.21.15.78:0/2965695894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c70115680 con 0x7f2c701091f0 2024-09-16T01:21:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.499+0000 7f2c6effd640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800b450 tx=0x7f2c5800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.504+0000 7f2c67fff640 1 -- 172.21.15.78:0/2965695894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58015020 con 0x7f2c701091f0 2024-09-16T01:21:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.504+0000 7f2c67fff640 1 -- 172.21.15.78:0/2965695894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c58011d70 con 0x7f2c701091f0 2024-09-16T01:21:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.504+0000 7f2c67fff640 1 -- 172.21.15.78:0/2965695894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5800f340 con 0x7f2c701091f0 2024-09-16T01:21:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.505+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c701bc0d0 con 0x7f2c701091f0 2024-09-16T01:21:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.505+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c701bc5a0 con 0x7f2c701091f0 2024-09-16T01:21:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.514+0000 7f2c782d3740 1 Processor -- start 2024-09-16T01:21:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.515+0000 7f2c782d3740 1 -- start start 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a90aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a915770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a9180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f54a86f2f0 con 0x55f54a915cb0 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f54a817270 con 0x55f54a90aaf0 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c782d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f54a86b2b0 con 0x55f54a90b410 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c6e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a9180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c6e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a9180a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35308/0 (socket says 172.21.15.78:35308) 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c6e7fc640 1 -- 172.21.15.78:0/1866717933 learned_addr learned my addr 172.21.15.78:0/1866717933 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.516+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a90aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c67fff640 1 -- 172.21.15.78:0/2965695894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f2c58018020 con 0x7f2c701091f0 2024-09-16T01:21:29.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/1866717933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 msgr2=0x55f54a915770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a915770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/1866717933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 msgr2=0x55f54a9180a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a9180a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/1866717933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54a816340 con 0x55f54a90aaf0 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.517+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a90aed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098e0 tx=0x7f2c60009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b58abb88782d399 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.518+0000 7f2c657fa640 1 -- 172.21.15.78:0/1866717933 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60017070 con 0x55f54a90aaf0 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.518+0000 7f2c657fa640 1 -- 172.21.15.78:0/1866717933 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c60013470 con 0x55f54a90aaf0 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.518+0000 7f2c657fa640 1 -- 172.21.15.78:0/1866717933 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60012630 con 0x55f54a90aaf0 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.518+0000 7f2c67fff640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 0x7f2c5c0a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.518+0000 7f2c67fff640 1 -- 172.21.15.78:0/2965695894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f2c580cf9a0 con 0x7f2c701091f0 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 -- 172.21.15.78:0/1866717933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 msgr2=0x55f54a90aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a90aed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098e0 tx=0x7f2c60009da0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 -- 172.21.15.78:0/1866717933 shutdown_connections 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a9180a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a915770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1866717933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a90aed0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.520+0000 7f2c782d3740 1 -- 172.21.15.78:0/1866717933 >> 172.21.15.78:0/1866717933 conn(0x55f54a907060 msgr2=0x55f54a907440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.521+0000 7f2c6effd640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 0x7f2c5c0a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.522+0000 7f2c782d3740 1 -- 172.21.15.78:0/1866717933 shutdown_connections 2024-09-16T01:21:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.522+0000 7f2c782d3740 1 -- 172.21.15.78:0/1866717933 wait complete. 2024-09-16T01:21:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.523+0000 7f2c782d3740 1 -- start start 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.523+0000 7f2c782d3740 1 Processor -- start 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.523+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 0x55f54a912c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.523+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a913180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 0x55f54a912c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 0x55f54a912c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35312/0 (socket says 172.21.15.78:35312) 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c6effd640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 0x7f2c5c0a5b50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55f54a907ec0 tx=0x7f2c58010fb0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/1359865976 learned_addr learned my addr 172.21.15.78:0/1359865976 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a915cb0 0x55f54a9136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a913180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f54a918880 con 0x55f54a90aaf0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f54a7f8290 con 0x55f54a915cb0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.524+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f54a918a00 con 0x55f54a90b410 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c782d3740 10 client.0 _ll_register_callbacks cb 0x55f54a829140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a915cb0 0x55f54a9136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 -- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a915cb0 msgr2=0x55f54a9136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a915cb0 0x55f54a9136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 -- 172.21.15.78:0/1359865976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 msgr2=0x55f54a912c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 0x55f54a912c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54a582210 con 0x55f54a90b410 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c567fc640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a913180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55f54a90d690 tx=0x7f2c680316b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=268873a5769fc92a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.525+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68048020 con 0x55f54a90b410 2024-09-16T01:21:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f2c68041050 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c60013470 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2c68048020 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c782d3740 10 client.25963 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f54a64e620 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.526+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c60013470 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2c68043070 con 0x55f54a90b410 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 10 client.25963 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 10 client.25963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 20 client.25963 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a925c00 0x55f54a928050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 4 client.25963 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.527+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f54a87ca20 con 0x55f54a925c00 2024-09-16T01:21:29.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:29 smithi078 ceph-mon[30937]: pgmap v333: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 630 op/s 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.528+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a925c00 0x55f54a928050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.529+0000 7f2c567fc640 10 client.25963 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.529+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a925c00 0x55f54a928050 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38073232e0c55883 server_cookie=1dbfe42b4b3a973e in_seq=0 out_seq=0 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.530+0000 7f2c567fc640 1 -- 172.21.15.78:0/1359865976 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f54a87ca20 con 0x55f54a925c00 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.530+0000 7f2c567fc640 10 client.25963 handle_command_reply: tid=0 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.530+0000 7f2c782d3740 1 client.25963 shutdown 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 msgr2=0x55f54a913180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a913180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55f54a90d690 tx=0x7f2c680316b0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 shutdown_connections 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a925c00 0x55f54a928050 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c567fc640 0 client.25963 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a915cb0 0x55f54a9136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90b410 0x55f54a913180 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55f54a90d690 tx=0x7f2c680316b0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 --2- 172.21.15.78:0/1359865976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a90aaf0 0x55f54a912c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.531+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 >> 172.21.15.78:0/1359865976 conn(0x55f54a907060 msgr2=0x55f54a90cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.532+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 shutdown_connections 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.532+0000 7f2c782d3740 1 -- 172.21.15.78:0/1359865976 wait complete. 2024-09-16T01:21:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.532+0000 7f2c782d3740 20 client.25963 trim_cache size 0 max 16384 2024-09-16T01:21:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:29 smithi193 ceph-mon[35701]: pgmap v333: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.5 MiB/s wr, 630 op/s 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 Processor -- start 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 -- start start 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 0x55f54a9134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a914bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a925c00 0x55f54a92d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c92f0 con 0x55f54a915cb0 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c6bc0 con 0x55f54a925c00 2024-09-16T01:21:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c782d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c49b0 con 0x55f54a90aaf0 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c6e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a914bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c6e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a914bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35324/0 (socket says 172.21.15.78:35324) 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c6e7fc640 1 -- 172.21.15.78:0/3539710330 learned_addr learned my addr 172.21.15.78:0/3539710330 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.556+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 0x55f54a9134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a925c00 0x55f54a92d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/3539710330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a925c00 msgr2=0x55f54a92d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a925c00 0x55f54a92d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/3539710330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 msgr2=0x55f54a914bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a914bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/3539710330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54a9111b0 con 0x55f54a90aaf0 2024-09-16T01:21:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.557+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 0x55f54a9134f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2c60013dd0 tx=0x7f2c6001f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fdc13b133b3a345 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c64ff9640 1 -- 172.21.15.78:0/3539710330 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60017040 con 0x55f54a90aaf0 2024-09-16T01:21:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c64ff9640 1 -- 172.21.15.78:0/3539710330 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6002f070 con 0x55f54a90aaf0 2024-09-16T01:21:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c64ff9640 1 -- 172.21.15.78:0/3539710330 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60029b80 con 0x55f54a90aaf0 2024-09-16T01:21:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 -- 172.21.15.78:0/3539710330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 msgr2=0x55f54a9134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 0x55f54a9134f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2c60013dd0 tx=0x7f2c6001f040 comp rx=0 tx=0).stop 2024-09-16T01:21:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 -- 172.21.15.78:0/3539710330 shutdown_connections 2024-09-16T01:21:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a925c00 0x55f54a92d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a915cb0 0x55f54a914bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 --2- 172.21.15.78:0/3539710330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a90aaf0 0x55f54a9134f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.558+0000 7f2c782d3740 1 -- 172.21.15.78:0/3539710330 >> 172.21.15.78:0/3539710330 conn(0x55f54a907060 msgr2=0x55f54a9144a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- 172.21.15.78:0/3539710330 shutdown_connections 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- 172.21.15.78:0/3539710330 wait complete. 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- start start 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 Processor -- start 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a91e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a915cb0 0x55f54a90faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c5080 con 0x55f54a925c00 2024-09-16T01:21:29.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c64f0 con 0x55f54a90aaf0 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c701c5750 con 0x55f54a915cb0 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.559+0000 7f2c782d3740 10 client.0 _ll_register_callbacks cb 0x7f2c68009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35340/0 (socket says 172.21.15.78:35340) 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/2335388756 learned_addr learned my addr 172.21.15.78:0/2335388756 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a91e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/2335388756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a915cb0 msgr2=0x55f54a90faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a915cb0 0x55f54a90faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 msgr2=0x55f54a91e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a91e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 -- 172.21.15.78:0/2335388756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54a98f930 con 0x55f54a925c00 2024-09-16T01:21:29.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c6f7fe640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x55f54a90f310 tx=0x7f2c60029760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8745f9e2691b7946 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c567fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.560+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60017070 con 0x55f54a925c00 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6002f070 con 0x55f54a925c00 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f2c60005c50 con 0x55f54a925c00 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2c60005e50 con 0x55f54a925c00 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 10 client.16587 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f54a990a80 con 0x55f54a925c00 2024-09-16T01:21:29.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6002f070 con 0x55f54a925c00 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2c60029070 con 0x55f54a925c00 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 10 client.16587 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 20 client.16587 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a990e20 0x55f54a993210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 4 client.16587 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.561+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f54a876d70 con 0x55f54a990e20 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.562+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a990e20 0x55f54a993210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.562+0000 7f2c567fc640 10 client.16587 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.562+0000 7f2c6e7fc640 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a990e20 0x55f54a993210 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416d0ca036fccff5 server_cookie=48bb3681d3035722 in_seq=0 out_seq=0 2024-09-16T01:21:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.571+0000 7f2c567fc640 1 -- 172.21.15.78:0/2335388756 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f54a876d70 con 0x55f54a990e20 2024-09-16T01:21:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.571+0000 7f2c567fc640 10 client.16587 handle_command_reply: tid=0 2024-09-16T01:21:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.571+0000 7f2c782d3740 1 client.16587 shutdown 2024-09-16T01:21:29.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.571+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 msgr2=0x55f54a9101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.571+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x55f54a90f310 tx=0x7f2c60029760 comp rx=0 tx=0).stop 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 shutdown_connections 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f54a990e20 0x55f54a993210 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c567fc640 0 client.16587 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54a925c00 0x55f54a9101e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f54a915cb0 0x55f54a90faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 --2- 172.21.15.78:0/2335388756 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f54a90aaf0 0x55f54a91e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.572+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 >> 172.21.15.78:0/2335388756 conn(0x55f54a907060 msgr2=0x55f54a90cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.573+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 shutdown_connections 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.573+0000 7f2c782d3740 1 -- 172.21.15.78:0/2335388756 wait complete. 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.573+0000 7f2c782d3740 20 client.16587 trim_cache size 0 max 16384 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:21:29.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:21:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 msgr2=0x7f2c5c0a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 0x7f2c5c0a5b50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55f54a907ec0 tx=0x7f2c58010fb0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 msgr2=0x7f2c7011a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2c5800b450 tx=0x7f2c5800b910 comp rx=0 tx=0).stop 2024-09-16T01:21:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 shutdown_connections 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f2c5c0a3690 0x7f2c5c0a5b50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c7011b140 0x7f2c70114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c701095d0 0x7f2c70114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 --2- 172.21.15.78:0/2965695894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c701091f0 0x7f2c7011a900 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 >> 172.21.15.78:0/2965695894 conn(0x7f2c7006ce00 msgr2=0x7f2c70070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 shutdown_connections 2024-09-16T01:21:29.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:29.574+0000 7f2c75531640 1 -- 172.21.15.78:0/2965695894 wait complete. 2024-09-16T01:21:30.941 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 615.0 s, 531.2 tps, lat 198.849 ms stddev 271.252 2024-09-16T01:21:31.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:31 smithi078 ceph-mon[30937]: pgmap v334: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 665 op/s 2024-09-16T01:21:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:31 smithi193 ceph-mon[35701]: pgmap v334: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 665 op/s 2024-09-16T01:21:31.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:31 smithi148 ceph-mon[35030]: pgmap v334: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 665 op/s 2024-09-16T01:21:33.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:33 smithi078 ceph-mon[30937]: pgmap v335: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-16T01:21:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:33 smithi193 ceph-mon[35701]: pgmap v335: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-16T01:21:33.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:33 smithi148 ceph-mon[35030]: pgmap v335: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-16T01:21:35.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:35 smithi078 ceph-mon[30937]: pgmap v336: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 615 op/s 2024-09-16T01:21:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:35 smithi193 ceph-mon[35701]: pgmap v336: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 615 op/s 2024-09-16T01:21:35.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:35 smithi148 ceph-mon[35030]: pgmap v336: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.0 MiB/s wr, 615 op/s 2024-09-16T01:21:35.828 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T01:21:35.828 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-16T01:21:35.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:21:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 620.0 s, 810.3 tps, lat 124.702 ms stddev 156.816 2024-09-16T01:21:36.345 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:21:37.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.084+0000 7fdd2fd69640 1 -- 172.21.15.78:0/4217743328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 msgr2=0x7fdd28110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.084+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/4217743328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd28110250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009be0 tx=0x7fdd2002a710 comp rx=0 tx=0).stop 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.086+0000 7fdd2fd69640 1 -- 172.21.15.78:0/4217743328 shutdown_connections 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.086+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/4217743328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd2811b140 0x7fdd2811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.086+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/4217743328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28109830 0x7fdd28109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.086+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/4217743328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd28110250 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.086+0000 7fdd2fd69640 1 -- 172.21.15.78:0/4217743328 >> 172.21.15.78:0/4217743328 conn(0x7fdd2806e140 msgr2=0x7fdd2806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:37.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.089+0000 7fdd2fd69640 1 -- 172.21.15.78:0/4217743328 shutdown_connections 2024-09-16T01:21:37.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.092+0000 7fdd2fd69640 1 -- 172.21.15.78:0/4217743328 wait complete. 2024-09-16T01:21:37.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.092+0000 7fdd2fd69640 1 Processor -- start 2024-09-16T01:21:37.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.093+0000 7fdd2fd69640 1 -- start start 2024-09-16T01:21:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.093+0000 7fdd2fd69640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2fd69640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd28109830 0x7fdd281a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2fd69640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd2811b140 0x7fdd281abc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2fd69640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28120e30 con 0x7fdd2811b140 2024-09-16T01:21:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2fd69640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28120cb0 con 0x7fdd28109830 2024-09-16T01:21:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2fd69640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28120fb0 con 0x7fdd28109450 2024-09-16T01:21:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.094+0000 7fdd2db05640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:37.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.095+0000 7fdd2d304640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd28109830 0x7fdd281a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:37.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.095+0000 7fdd2db05640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:33734/0 (socket says 172.21.15.78:33734) 2024-09-16T01:21:37.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.095+0000 7fdd2db05640 1 -- 172.21.15.78:0/573314621 learned_addr learned my addr 172.21.15.78:0/573314621 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:37.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.099+0000 7fdd2e306640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd2811b140 0x7fdd281abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.100+0000 7fdd2db05640 1 -- 172.21.15.78:0/573314621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd28109830 msgr2=0x7fdd281a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.100+0000 7fdd2db05640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd28109830 0x7fdd281a78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.100+0000 7fdd2db05640 1 -- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd2811b140 msgr2=0x7fdd281abc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.100+0000 7fdd2db05640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd2811b140 0x7fdd281abc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.100+0000 7fdd2db05640 1 -- 172.21.15.78:0/573314621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd281ac340 con 0x7fdd28109450 2024-09-16T01:21:37.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.108+0000 7fdd2db05640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f80 tx=0x7fdd20030980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:37.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.110+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20041020 con 0x7fdd28109450 2024-09-16T01:21:37.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.110+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd20030cf0 con 0x7fdd28109450 2024-09-16T01:21:37.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.110+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20036040 con 0x7fdd28109450 2024-09-16T01:21:37.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.113+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd281ac5d0 con 0x7fdd28109450 2024-09-16T01:21:37.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.113+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd281aca30 con 0x7fdd28109450 2024-09-16T01:21:37.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.116+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd281a86f0 con 0x7fdd28109450 2024-09-16T01:21:37.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.119+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fdd20044020 con 0x7fdd28109450 2024-09-16T01:21:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.120+0000 7fdd1effd640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 0x7fdd140a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.121+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fdd200f0db0 con 0x7fdd28109450 2024-09-16T01:21:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.123+0000 7fdd2d304640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 0x7fdd140a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.123+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd2004b070 con 0x7fdd28109450 2024-09-16T01:21:37.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.130+0000 7fdd2d304640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 0x7fdd140a5a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdd2811d8f0 tx=0x7fdd24009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:37.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:37 smithi078 ceph-mon[30937]: pgmap v337: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-09-16T01:21:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.507+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd2810b3a0 con 0x7fdd28109450 2024-09-16T01:21:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.508+0000 7fdd1effd640 1 -- 172.21.15.78:0/573314621 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fdd200bd4d0 con 0x7fdd28109450 2024-09-16T01:21:37.511 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:21:37.511 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:21:37.511 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:21:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.514+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 msgr2=0x7fdd140a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.514+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 0x7fdd140a5a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdd2811d8f0 tx=0x7fdd24009260 comp rx=0 tx=0).stop 2024-09-16T01:21:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.514+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 msgr2=0x7fdd281a7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.514+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f80 tx=0x7fdd20030980 comp rx=0 tx=0).stop 2024-09-16T01:21:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 shutdown_connections 2024-09-16T01:21:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd140a35c0 0x7fdd140a5a80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd2811b140 0x7fdd281abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd28109830 0x7fdd281a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 --2- 172.21.15.78:0/573314621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd28109450 0x7fdd281a7390 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.515+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 >> 172.21.15.78:0/573314621 conn(0x7fdd2806e140 msgr2=0x7fdd2811be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.516+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 shutdown_connections 2024-09-16T01:21:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:37.518+0000 7fdd2fd69640 1 -- 172.21.15.78:0/573314621 wait complete. 2024-09-16T01:21:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:37 smithi193 ceph-mon[35701]: pgmap v337: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-09-16T01:21:37.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:37 smithi148 ceph-mon[35030]: pgmap v337: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-09-16T01:21:38.493 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/573314621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:38.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/573314621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:38.550 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-16T01:21:38.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-16T01:21:38.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/573314621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:39.130 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:21:39.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:39 smithi078 ceph-mon[30937]: pgmap v338: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 654 op/s 2024-09-16T01:21:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:39 smithi193 ceph-mon[35701]: pgmap v338: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 654 op/s 2024-09-16T01:21:39.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:39 smithi148 ceph-mon[35030]: pgmap v338: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 654 op/s 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 -- 172.21.15.78:0/4007136939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc100e20 msgr2=0x7fb5bc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/4007136939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc100e20 0x7fb5bc101200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009930 tx=0x7fb5b002a4d0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 -- 172.21.15.78:0/4007136939 shutdown_connections 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/4007136939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/4007136939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 0x7fb5bc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/4007136939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc100e20 0x7fb5bc101200 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.903+0000 7fb5c2b79640 1 -- 172.21.15.78:0/4007136939 >> 172.21.15.78:0/4007136939 conn(0x7fb5bc0fc7d0 msgr2=0x7fb5bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:39.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- 172.21.15.78:0/4007136939 shutdown_connections 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- 172.21.15.78:0/4007136939 wait complete. 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 Processor -- start 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- start start 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc100e20 0x7fb5bc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 0x7fb5bc198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc110fd0 con 0x7fb5bc102570 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc110e50 con 0x7fb5bc102ef0 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.904+0000 7fb5c2b79640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc111150 con 0x7fb5bc100e20 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:38524/0 (socket says 172.21.15.78:38524) 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 -- 172.21.15.78:0/1777840796 learned_addr learned my addr 172.21.15.78:0/1777840796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:39.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c0915640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc100e20 0x7fb5bc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 0x7fb5bc198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 -- 172.21.15.78:0/1777840796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc100e20 msgr2=0x7fb5bc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc100e20 0x7fb5bc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 -- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 msgr2=0x7fb5bc198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 0x7fb5bc198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 -- 172.21.15.78:0/1777840796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc19d520 con 0x7fb5bc102ef0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400eb10 tx=0x7fb5b400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5b9ffb640 1 -- 172.21.15.78:0/1777840796 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4004b20 con 0x7fb5bc102ef0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5b9ffb640 1 -- 172.21.15.78:0/1777840796 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b4004cc0 con 0x7fb5bc102ef0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc19d810 con 0x7fb5bc102ef0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.905+0000 7fb5b9ffb640 1 -- 172.21.15.78:0/1777840796 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4010760 con 0x7fb5bc102ef0 2024-09-16T01:21:39.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.906+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc1a50f0 con 0x7fb5bc102ef0 2024-09-16T01:21:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.908+0000 7fb5b9ffb640 1 -- 172.21.15.78:0/1777840796 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb5b4005020 con 0x7fb5bc102ef0 2024-09-16T01:21:39.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 Processor -- start 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 -- start start 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 0x55a804c46070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 0x55a804c5b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a804baa160 con 0x55a804c45c30 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a804b51720 con 0x55a804c590d0 2024-09-16T01:21:39.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.909+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a804ba6120 con 0x55a804c46dd0 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.910+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.910+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 0x55a804c5b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.910+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:33760/0 (socket says 172.21.15.78:33760) 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.910+0000 7fb5bbfff640 1 -- 172.21.15.78:0/2264165587 learned_addr learned my addr 172.21.15.78:0/2264165587 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.910+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 0x55a804c46070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5c0915640 1 -- 172.21.15.78:0/2264165587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 msgr2=0x55a804c456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5c0915640 1 -- 172.21.15.78:0/2264165587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 msgr2=0x55a804c5b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 0x55a804c5b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5b9ffb640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 0x7fb5ac0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5c0915640 1 -- 172.21.15.78:0/2264165587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a80491bba0 con 0x55a804c45c30 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 0x55a804c5b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5b9ffb640 1 -- 172.21.15.78:0/1777840796 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fb5b40d07b0 con 0x7fb5bc102ef0 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.911+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 0x7fb5ac0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.912+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 0x55a804c46070 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002a9f0 tx=0x7fb5b0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878f2e66f424755e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.912+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2264165587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0035040 con 0x55a804c45c30 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.912+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 0x7fb5ac0a5870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a804c42ed0 tx=0x7fb5a8015040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.912+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2264165587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b0005520 con 0x55a804c45c30 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.912+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2264165587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0004e10 con 0x55a804c45c30 2024-09-16T01:21:39.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2264165587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 msgr2=0x55a804c46070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 0x55a804c46070 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002a9f0 tx=0x7fb5b0005de0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2264165587 shutdown_connections 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c590d0 0x55a804c5b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c45c30 0x55a804c46070 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2264165587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2264165587 >> 172.21.15.78:0/2264165587 conn(0x55a804c41de0 msgr2=0x55a804c421c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2264165587 shutdown_connections 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.913+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2264165587 wait complete. 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 -- start start 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 Processor -- start 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c45c30 0x55a804c52d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c46dd0 0x55a804c58a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a804c4cc40 con 0x55a804c590d0 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a804b32900 con 0x55a804c46dd0 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a804c4cdc0 con 0x55a804c45c30 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c4f52740 10 client.0 _ll_register_callbacks cb 0x55a804b30780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41784/0 (socket says 172.21.15.78:41784) 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 -- 172.21.15.78:0/1194823381 learned_addr learned my addr 172.21.15.78:0/1194823381 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c0915640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c45c30 0x55a804c52d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 -- 172.21.15.78:0/1194823381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c45c30 msgr2=0x55a804c52d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c45c30 0x55a804c52d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 -- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c46dd0 msgr2=0x55a804c58a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c46dd0 0x55a804c58a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.914+0000 7fb5c1116640 1 -- 172.21.15.78:0/1194823381 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a804b72c50 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb5c1116640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40cdbc0 tx=0x7fb5b409c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d84810bf66f3e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb59bfff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40d0580 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5b0005520 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fb5b4005310 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5b4005550 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb5c4f52740 10 client.16605 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a804c703c0 con 0x55a804c590d0 2024-09-16T01:21:39.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.915+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b0005520 con 0x55a804c590d0 2024-09-16T01:21:39.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb5b40b0bc0 con 0x55a804c590d0 2024-09-16T01:21:39.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 10 client.16605 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:39.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 10 client.16605 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:21:39.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 20 client.16605 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:39.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804c71880 0x55a804c73d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 4 client.16605 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:39.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.916+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a804bb7890 con 0x55a804c71880 2024-09-16T01:21:39.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.918+0000 7fb5c0915640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804c71880 0x55a804c73d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.918+0000 7fb5c0915640 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804c71880 0x55a804c73d40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=601d4ed6c7d23856 server_cookie=c6f4e427538d44d7 in_seq=0 out_seq=0 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.918+0000 7fb59bfff640 10 client.16605 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb59bfff640 1 -- 172.21.15.78:0/1194823381 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a804bb7890 con 0x55a804c71880 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb59bfff640 10 client.16605 handle_command_reply: tid=0 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 client.16605 shutdown 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 msgr2=0x55a804c53390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb5b40cdbc0 tx=0x7fb5b409c1e0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 shutdown_connections 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804c71880 0x55a804c73d40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c53390 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c46dd0 0x55a804c58a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/1194823381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c45c30 0x55a804c52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb59bfff640 0 client.16605 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 >> 172.21.15.78:0/1194823381 conn(0x55a804c41de0 msgr2=0x55a804c47b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 shutdown_connections 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 1 -- 172.21.15.78:0/1194823381 wait complete. 2024-09-16T01:21:39.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.919+0000 7fb5c4f52740 20 client.16605 trim_cache size 0 max 16384 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 Processor -- start 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 -- start start 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c79610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1af8e0 con 0x55a804c590d0 2024-09-16T01:21:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1ad1b0 con 0x55a804c45c30 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1ab020 con 0x55a804c46dd0 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c1116640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41798/0 (socket says 172.21.15.78:41798) 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c1116640 1 -- 172.21.15.78:0/500432439 learned_addr learned my addr 172.21.15.78:0/500432439 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c79610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.940+0000 7fb5c0915640 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 -- 172.21.15.78:0/500432439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 msgr2=0x55a804c79610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c79610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 -- 172.21.15.78:0/500432439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 msgr2=0x55a804c589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 -- 172.21.15.78:0/500432439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a804c46640 con 0x55a804c590d0 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5c1116640 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb5b409c3a0 tx=0x7fb5b409c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806f17ae793d3194 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5a67fc640 1 -- 172.21.15.78:0/500432439 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4083070 con 0x55a804c590d0 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5a67fc640 1 -- 172.21.15.78:0/500432439 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b409c6a0 con 0x55a804c590d0 2024-09-16T01:21:39.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.941+0000 7fb5a67fc640 1 -- 172.21.15.78:0/500432439 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40a5310 con 0x55a804c590d0 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- 172.21.15.78:0/500432439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 msgr2=0x55a804c79b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb5b409c3a0 tx=0x7fb5b409c310 comp rx=0 tx=0).stop 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- 172.21.15.78:0/500432439 shutdown_connections 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c79b50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c79610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/500432439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- 172.21.15.78:0/500432439 >> 172.21.15.78:0/500432439 conn(0x55a804c41de0 msgr2=0x55a804c48890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- 172.21.15.78:0/500432439 shutdown_connections 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- 172.21.15.78:0/500432439 wait complete. 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 -- start start 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.942+0000 7fb5c4f52740 1 Processor -- start 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1ab670 con 0x55a804c590d0 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1acae0 con 0x55a804c45c30 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc1abd40 con 0x55a804c46dd0 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c4f52740 10 client.0 _ll_register_callbacks cb 0x7fb5b40057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:33776/0 (socket says 172.21.15.78:33776) 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 -- 172.21.15.78:0/2527640407 learned_addr learned my addr 172.21.15.78:0/2527640407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c85340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5c1116640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 -- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 msgr2=0x55a804c85340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c85340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 -- 172.21.15.78:0/2527640407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 msgr2=0x55a804c497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c497b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 -- 172.21.15.78:0/2527640407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a804c4b440 con 0x55a804c46dd0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb5bbfff640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55a804c4d930 tx=0x7fb5a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e040acc7eb28786 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.943+0000 7fb59bfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8021430 con 0x55a804c46dd0 2024-09-16T01:21:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5b409c6a0 con 0x55a804c46dd0 2024-09-16T01:21:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fb5a801c070 con 0x55a804c46dd0 2024-09-16T01:21:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb5a8039050 con 0x55a804c46dd0 2024-09-16T01:21:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 10 client.25990 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a804cdd8f0 con 0x55a804c46dd0 2024-09-16T01:21:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b409c6a0 con 0x55a804c46dd0 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb5a802e620 con 0x55a804c46dd0 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 10 client.25990 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 10 client.25990 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 20 client.25990 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804cde190 0x55a804ce05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 4 client.25990 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.944+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x55a804c3d410 con 0x55a804cde190 2024-09-16T01:21:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.945+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804cde190 0x55a804ce05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.946+0000 7fb5c0915640 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804cde190 0x55a804ce05b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=486ac5e39221ef24 server_cookie=a0ac233187c732cf in_seq=0 out_seq=0 2024-09-16T01:21:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:39.946+0000 7fb59bfff640 10 client.25990 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:40.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb59bfff640 1 -- 172.21.15.78:0/2527640407 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a804c3d410 con 0x55a804cde190 2024-09-16T01:21:40.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb59bfff640 10 client.25990 handle_command_reply: tid=0 2024-09-16T01:21:40.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 client.25990 shutdown 2024-09-16T01:21:40.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 msgr2=0x55a804c85880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55a804c4d930 tx=0x7fb5a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 shutdown_connections 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a804cde190 0x55a804ce05b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a804c590d0 0x55a804c497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a804c46dd0 0x55a804c85880 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb59bfff640 0 client.25990 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 --2- 172.21.15.78:0/2527640407 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a804c45c30 0x55a804c85340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 >> 172.21.15.78:0/2527640407 conn(0x55a804c41de0 msgr2=0x55a804c472e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 shutdown_connections 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 1 -- 172.21.15.78:0/2527640407 wait complete. 2024-09-16T01:21:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.499+0000 7fb5c4f52740 20 client.25990 trim_cache size 0 max 16384 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 38, 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "dde09c3c": { 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 38, 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:21:40.504 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 141.8, 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:21:40.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:21:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:21:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:21:40.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:21:40.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:21:40.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 msgr2=0x7fb5ac0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 0x7fb5ac0a5870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a804c42ed0 tx=0x7fb5a8015040 comp rx=0 tx=0).stop 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 msgr2=0x7fb5bc19ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400eb10 tx=0x7fb5b400c2d0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 shutdown_connections 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb5ac0a33b0 0x7fb5ac0a5870 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5bc102ef0 0x7fb5bc19ce40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5bc102570 0x7fb5bc198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 --2- 172.21.15.78:0/1777840796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5bc100e20 0x7fb5bc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:40.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 >> 172.21.15.78:0/1777840796 conn(0x7fb5bc0fc7d0 msgr2=0x7fb5bc06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:40.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 shutdown_connections 2024-09-16T01:21:40.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:40.500+0000 7fb5c2b79640 1 -- 172.21.15.78:0/1777840796 wait complete. 2024-09-16T01:21:40.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 625.0 s, 517.0 tps, lat 159.181 ms stddev 193.442 2024-09-16T01:21:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:41 smithi078 ceph-mon[30937]: pgmap v339: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.7 MiB/s wr, 674 op/s 2024-09-16T01:21:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:41 smithi193 ceph-mon[35701]: pgmap v339: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.7 MiB/s wr, 674 op/s 2024-09-16T01:21:41.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:41 smithi148 ceph-mon[35030]: pgmap v339: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.7 MiB/s wr, 674 op/s 2024-09-16T01:21:42.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 -- 172.21.15.78:0/1552266965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde64102d40 msgr2=0x7fde64103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/1552266965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde64102d40 0x7fde64103120 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fde54009900 tx=0x7fde5402a480 comp rx=0 tx=0).stop 2024-09-16T01:21:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 -- 172.21.15.78:0/1552266965 shutdown_connections 2024-09-16T01:21:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/1552266965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde6410c0d0 0x7fde6410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/1552266965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64103660 0x7fde6410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/1552266965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde64102d40 0x7fde64103120 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 -- 172.21.15.78:0/1552266965 >> 172.21.15.78:0/1552266965 conn(0x7fde640fc6f0 msgr2=0x7fde640feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 -- 172.21.15.78:0/1552266965 shutdown_connections 2024-09-16T01:21:42.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.671+0000 7fde6aaf1640 1 -- 172.21.15.78:0/1552266965 wait complete. 2024-09-16T01:21:42.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 Processor -- start 2024-09-16T01:21:42.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 -- start start 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 0x7fde64198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde64103660 0x7fde64198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde64111ed0 con 0x7fde6410c0d0 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde64111d50 con 0x7fde64102d40 2024-09-16T01:21:42.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6aaf1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde64112050 con 0x7fde64103660 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6908e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6908e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41830/0 (socket says 172.21.15.78:41830) 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.672+0000 7fde6908e640 1 -- 172.21.15.78:0/3099902866 learned_addr learned my addr 172.21.15.78:0/3099902866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde63fff640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde64103660 0x7fde64198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6888d640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 0x7fde64198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 -- 172.21.15.78:0/3099902866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde64103660 msgr2=0x7fde64198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde64103660 0x7fde64198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 -- 172.21.15.78:0/3099902866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 msgr2=0x7fde64198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 0x7fde64198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 -- 172.21.15.78:0/3099902866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6419d300 con 0x7fde6410c0d0 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6888d640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 0x7fde64198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:21:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6908e640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00eaa0 tx=0x7fde5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde61ffb640 1 -- 172.21.15.78:0/3099902866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00cdb0 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde61ffb640 1 -- 172.21.15.78:0/3099902866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c005020 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6419d5f0 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde61ffb640 1 -- 172.21.15.78:0/3099902866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0116b0 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.673+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde641a4e80 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.675+0000 7fde6d890740 1 Processor -- start 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.675+0000 7fde61ffb640 1 -- 172.21.15.78:0/3099902866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fde5c005560 con 0x7fde6410c0d0 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 -- start start 2024-09-16T01:21:42.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45022ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c450300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560c44f872f0 con 0x560c45023410 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560c44f2f270 con 0x560c4502dcb0 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6d890740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c44f832b0 con 0x560c45022af0 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6888d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6888d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41844/0 (socket says 172.21.15.78:41844) 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde6888d640 1 -- 172.21.15.78:0/470219973 learned_addr learned my addr 172.21.15.78:0/470219973 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde61ffb640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 0x7fde4c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.676+0000 7fde63fff640 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45022ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde63fff640 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c450300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6908e640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 0x7fde4c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 -- 172.21.15.78:0/470219973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 msgr2=0x560c45022ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45022ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 -- 172.21.15.78:0/470219973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 msgr2=0x560c450300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c450300a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 -- 172.21.15.78:0/470219973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c44f2e340 con 0x560c45023410 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde61ffb640 1 -- 172.21.15.78:0/3099902866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fde5c0d0300 con 0x7fde6410c0d0 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6888d640 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fde54031550 tx=0x7fde54032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca5438b9959ffc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde6908e640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 0x7fde4c0a59a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fde5c098650 tx=0x7fde5c06be10 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde4affd640 1 -- 172.21.15.78:0/470219973 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5403f070 con 0x560c45023410 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde4affd640 1 -- 172.21.15.78:0/470219973 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde54004030 con 0x560c45023410 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.677+0000 7fde4affd640 1 -- 172.21.15.78:0/470219973 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54035070 con 0x560c45023410 2024-09-16T01:21:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- 172.21.15.78:0/470219973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 msgr2=0x560c4502d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fde54031550 tx=0x7fde54032c90 comp rx=0 tx=0).stop 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- 172.21.15.78:0/470219973 shutdown_connections 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c450300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c4502d770 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 --2- 172.21.15.78:0/470219973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45022ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- 172.21.15.78:0/470219973 >> 172.21.15.78:0/470219973 conn(0x560c4501f060 msgr2=0x560c4501f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- 172.21.15.78:0/470219973 shutdown_connections 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- 172.21.15.78:0/470219973 wait complete. 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.678+0000 7fde6d890740 1 -- start start 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 Processor -- start 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45022af0 0x560c4502aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45023410 0x560c4502aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560c45030880 con 0x560c4502dcb0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560c44f10290 con 0x560c45022af0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c45030a00 con 0x560c45023410 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6d890740 10 client.0 _ll_register_callbacks cb 0x560c44f41140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41856/0 (socket says 172.21.15.78:41856) 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 -- 172.21.15.78:0/1595085328 learned_addr learned my addr 172.21.15.78:0/1595085328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde6888d640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45023410 0x560c4502aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45022af0 0x560c4502aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 -- 172.21.15.78:0/1595085328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45023410 msgr2=0x560c4502aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45023410 0x560c4502aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 -- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45022af0 msgr2=0x560c4502aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45022af0 0x560c4502aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 -- 172.21.15.78:0/1595085328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c44c9a210 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde63fff640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fde6410d510 tx=0x7fde5800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415083e394c4d3db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.679+0000 7fde43fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde58015e60 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fde58012b00 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde54004030 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde58028050 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 10 client.16623 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c44d66620 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde54004030 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fde5800f040 con 0x560c4502dcb0 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 10 client.16623 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 20 client.16623 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c4503db80 0x560c4503ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 4 client.16623 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c44f94a20 con 0x560c4503db80 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.680+0000 7fde6888d640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c4503db80 0x560c4503ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.681+0000 7fde43fff640 10 client.16623 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.681+0000 7fde6888d640 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c4503db80 0x560c4503ff90 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b903c4a268b8c971 server_cookie=8e37f3d03df2478c in_seq=0 out_seq=0 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde43fff640 1 -- 172.21.15.78:0/1595085328 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c44f94a20 con 0x560c4503db80 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde43fff640 10 client.16623 handle_command_reply: tid=0 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 client.16623 shutdown 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 msgr2=0x560c4502b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fde6410d510 tx=0x7fde5800eb80 comp rx=0 tx=0).stop 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 shutdown_connections 2024-09-16T01:21:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c4503db80 0x560c4503ff90 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c4502b530 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45023410 0x560c4502aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 --2- 172.21.15.78:0/1595085328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45022af0 0x560c4502aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde43fff640 0 client.16623 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 >> 172.21.15.78:0/1595085328 conn(0x560c4501f060 msgr2=0x560c45024d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 shutdown_connections 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 1 -- 172.21.15.78:0/1595085328 wait complete. 2024-09-16T01:21:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.682+0000 7fde6d890740 20 client.16623 trim_cache size 0 max 16384 2024-09-16T01:21:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.696+0000 7fde6d890740 1 Processor -- start 2024-09-16T01:21:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.696+0000 7fde6d890740 1 -- start start 2024-09-16T01:21:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.696+0000 7fde6d890740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c4502c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.696+0000 7fde6d890740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6d890740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c45045de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6d890740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde641b19e0 con 0x560c45023410 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6d890740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde641af2b0 con 0x560c4502dcb0 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6d890740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ad120 con 0x560c45022af0 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41864/0 (socket says 172.21.15.78:41864) 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 -- 172.21.15.78:0/4038024566 learned_addr learned my addr 172.21.15.78:0/4038024566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde63fff640 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c4502c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 -- 172.21.15.78:0/4038024566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 msgr2=0x560c4502c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c4502c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 -- 172.21.15.78:0/4038024566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 msgr2=0x560c45045de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c45045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 -- 172.21.15.78:0/4038024566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c4501ee90 con 0x560c45023410 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde6888d640 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde540051f0 tx=0x7fde54005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad1267c072960a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde4a7fc640 1 -- 172.21.15.78:0/4038024566 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54035040 con 0x560c45023410 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde4a7fc640 1 -- 172.21.15.78:0/4038024566 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde540044a0 con 0x560c45023410 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.697+0000 7fde4a7fc640 1 -- 172.21.15.78:0/4038024566 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5403f040 con 0x560c45023410 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- 172.21.15.78:0/4038024566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 msgr2=0x560c450458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde540051f0 tx=0x7fde54005220 comp rx=0 tx=0).stop 2024-09-16T01:21:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- 172.21.15.78:0/4038024566 shutdown_connections 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c4502dcb0 0x560c45045de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c45023410 0x560c450458a0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- 172.21.15.78:0/4038024566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c4502c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- 172.21.15.78:0/4038024566 >> 172.21.15.78:0/4038024566 conn(0x560c4501f060 msgr2=0x560c4501e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- 172.21.15.78:0/4038024566 shutdown_connections 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- 172.21.15.78:0/4038024566 wait complete. 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- start start 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 Processor -- start 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45032100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45023410 0x560c45032640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ad770 con 0x560c4502dcb0 2024-09-16T01:21:42.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fde641aebe0 con 0x560c45023410 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde641ade40 con 0x560c45022af0 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.698+0000 7fde6d890740 10 client.0 _ll_register_callbacks cb 0x7fde5800ed80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41874/0 (socket says 172.21.15.78:41874) 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 -- 172.21.15.78:0/2435684676 learned_addr learned my addr 172.21.15.78:0/2435684676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45032100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde6888d640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45023410 0x560c45032640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 -- 172.21.15.78:0/2435684676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 msgr2=0x560c45032100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45032100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 -- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45023410 msgr2=0x560c45032640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45023410 0x560c45032640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 -- 172.21.15.78:0/2435684676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c450a7db0 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde63fff640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fde5800efd0 tx=0x7fde58017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e6968d1db30b7e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde43fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde58015e60 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fde5801d070 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde540044a0 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde5803d050 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde6d890740 10 client.16629 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c450a9030 con 0x560c4502dcb0 2024-09-16T01:21:42.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.699+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde540044a0 con 0x560c4502dcb0 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fde58004080 con 0x560c4502dcb0 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 10 client.16629 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 20 client.16629 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c450a96f0 0x560c450abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 4 client.16629 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c44f8ed70 con 0x560c450a96f0 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6888d640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c450a96f0 0x560c450abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde6888d640 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c450a96f0 0x560c450abb10 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d092aa35585d3c server_cookie=ece6532a57268eec in_seq=0 out_seq=0 2024-09-16T01:21:42.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.700+0000 7fde43fff640 10 client.16629 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde43fff640 1 -- 172.21.15.78:0/2435684676 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c44f8ed70 con 0x560c450a96f0 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde43fff640 10 client.16629 handle_command_reply: tid=0 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 client.16629 shutdown 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 msgr2=0x560c450a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fde5800efd0 tx=0x7fde58017040 comp rx=0 tx=0).stop 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 shutdown_connections 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560c450a96f0 0x560c450abb10 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560c4502dcb0 0x560c450a7870 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560c45023410 0x560c45032640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde43fff640 0 client.16629 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:42.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 --2- 172.21.15.78:0/2435684676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c45022af0 0x560c45032100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 >> 172.21.15.78:0/2435684676 conn(0x560c4501f060 msgr2=0x560c45038b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 shutdown_connections 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 1 -- 172.21.15.78:0/2435684676 wait complete. 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.708+0000 7fde6d890740 20 client.16629 trim_cache size 0 max 16384 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:21:42.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:21:42.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 msgr2=0x7fde4c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 0x7fde4c0a59a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fde5c098650 tx=0x7fde5c06be10 comp rx=0 tx=0).stop 2024-09-16T01:21:42.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 msgr2=0x7fde6419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00eaa0 tx=0x7fde5c00ef60 comp rx=0 tx=0).stop 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 shutdown_connections 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fde4c0a34e0 0x7fde4c0a59a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6410c0d0 0x7fde6419cbd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde64103660 0x7fde64198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 --2- 172.21.15.78:0/3099902866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde64102d40 0x7fde64198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 >> 172.21.15.78:0/3099902866 conn(0x7fde640fc6f0 msgr2=0x7fde640fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 shutdown_connections 2024-09-16T01:21:42.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:42.709+0000 7fde6aaf1640 1 -- 172.21.15.78:0/3099902866 wait complete. 2024-09-16T01:21:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:43 smithi078 ceph-mon[30937]: pgmap v340: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.1 MiB/s wr, 625 op/s 2024-09-16T01:21:43.341 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dde09c3c', quiesce took 7.5 seconds. Will release after: 14.350000000000001 2024-09-16T01:21:43.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:43 smithi193 ceph-mon[35701]: pgmap v340: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.1 MiB/s wr, 625 op/s 2024-09-16T01:21:43.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:43 smithi148 ceph-mon[35030]: pgmap v340: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 9.1 MiB/s wr, 625 op/s 2024-09-16T01:21:45.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:45 smithi078 ceph-mon[30937]: pgmap v341: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 556 op/s 2024-09-16T01:21:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:45 smithi193 ceph-mon[35701]: pgmap v341: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 556 op/s 2024-09-16T01:21:45.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:45 smithi148 ceph-mon[35030]: pgmap v341: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.2 MiB/s wr, 556 op/s 2024-09-16T01:21:45.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 630.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:21:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:47 smithi078 ceph-mon[30937]: pgmap v342: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 7.9 MiB/s wr, 560 op/s 2024-09-16T01:21:47.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:47 smithi193 ceph-mon[35701]: pgmap v342: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 7.9 MiB/s wr, 560 op/s 2024-09-16T01:21:47.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:47 smithi148 ceph-mon[35030]: pgmap v342: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 7.9 MiB/s wr, 560 op/s 2024-09-16T01:21:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:49 smithi078 ceph-mon[30937]: pgmap v343: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.1 MiB/s wr, 388 op/s 2024-09-16T01:21:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:49 smithi193 ceph-mon[35701]: pgmap v343: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.1 MiB/s wr, 388 op/s 2024-09-16T01:21:49.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:49 smithi148 ceph-mon[35030]: pgmap v343: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.1 MiB/s wr, 388 op/s 2024-09-16T01:21:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 635.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:21:51.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:51 smithi078 ceph-mon[30937]: pgmap v344: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-09-16T01:21:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:51 smithi193 ceph-mon[35701]: pgmap v344: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-09-16T01:21:51.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:51 smithi148 ceph-mon[35030]: pgmap v344: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-09-16T01:21:53.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:53 smithi078 ceph-mon[30937]: pgmap v345: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 193 op/s 2024-09-16T01:21:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:53 smithi193 ceph-mon[35701]: pgmap v345: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 193 op/s 2024-09-16T01:21:53.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:53 smithi148 ceph-mon[35030]: pgmap v345: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 193 op/s 2024-09-16T01:21:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:55 smithi078 ceph-mon[30937]: pgmap v346: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-09-16T01:21:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:55 smithi193 ceph-mon[35701]: pgmap v346: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-09-16T01:21:55.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:55 smithi148 ceph-mon[35030]: pgmap v346: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.0 MiB/s wr, 100 op/s 2024-09-16T01:21:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 -- 172.21.15.78:0/3751761239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 msgr2=0x7f753c1030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 --2- 172.21.15.78:0/3751761239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c1030e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f752c009900 tx=0x7f752c02a480 comp rx=0 tx=0).stop 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 -- 172.21.15.78:0/3751761239 shutdown_connections 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 --2- 172.21.15.78:0/3751761239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c10c0d0 0x7f753c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 --2- 172.21.15.78:0/3751761239 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c103620 0x7f753c10bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 --2- 172.21.15.78:0/3751761239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c1030e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 -- 172.21.15.78:0/3751761239 >> 172.21.15.78:0/3751761239 conn(0x7f753c0fc6f0 msgr2=0x7f753c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 -- 172.21.15.78:0/3751761239 shutdown_connections 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 -- 172.21.15.78:0/3751761239 wait complete. 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.806+0000 7f7542942640 1 Processor -- start 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 -- start start 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c103620 0x7f753c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 0x7f753c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f753c111ed0 con 0x7f753c102d00 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f753c111d50 con 0x7f753c10c0d0 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7542942640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f753c112050 con 0x7f753c103620 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56556/0 (socket says 172.21.15.78:56556) 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 -- 172.21.15.78:0/1082750104 learned_addr learned my addr 172.21.15.78:0/1082750104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7533fff640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c103620 0x7f753c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7540edf640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 0x7f753c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 -- 172.21.15.78:0/1082750104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c103620 msgr2=0x7f753c198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c103620 0x7f753c198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 -- 172.21.15.78:0/1082750104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 msgr2=0x7f753c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 0x7f753c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 -- 172.21.15.78:0/1082750104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c19d330 con 0x7f753c102d00 2024-09-16T01:21:55.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7540edf640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 0x7f753c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f753bfff640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f752c006010 tx=0x7f752c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.807+0000 7f7539ffb640 1 -- 172.21.15.78:0/1082750104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c03f030 con 0x7f753c102d00 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.808+0000 7f7539ffb640 1 -- 172.21.15.78:0/1082750104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f752c003ce0 con 0x7f753c102d00 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.808+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c19d5c0 con 0x7f753c102d00 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.808+0000 7f7539ffb640 1 -- 172.21.15.78:0/1082750104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c035070 con 0x7f753c102d00 2024-09-16T01:21:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.808+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a4eb0 con 0x7f753c102d00 2024-09-16T01:21:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.810+0000 7f7539ffb640 1 -- 172.21.15.78:0/1082750104 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f752c042030 con 0x7f753c102d00 2024-09-16T01:21:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.810+0000 7f75456d9740 1 Processor -- start 2024-09-16T01:21:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.810+0000 7f75456d9740 1 -- start start 2024-09-16T01:21:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b282d0 0x5623e7b36b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5623e7a8c2f0 con 0x5623e7b279b0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5623e7a34270 con 0x5623e7b282d0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75456d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5623e7a882b0 con 0x5623e7b370c0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7539ffb640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 0x7f75280a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56562/0 (socket says 172.21.15.78:56562) 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 -- 172.21.15.78:0/825675360 learned_addr learned my addr 172.21.15.78:0/825675360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7533fff640 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 0x7f75280a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7539ffb640 1 -- 172.21.15.78:0/1082750104 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f752c0f0720 con 0x7f753c102d00 2024-09-16T01:21:55.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f753bfff640 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b282d0 0x5623e7b36b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 -- 172.21.15.78:0/825675360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 msgr2=0x5623e7b394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b394b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 -- 172.21.15.78:0/825675360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b282d0 msgr2=0x5623e7b36b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b282d0 0x5623e7b36b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 -- 172.21.15.78:0/825675360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e7a33340 con 0x5623e7b279b0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f7540edf640 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7534013aa0 tx=0x7f7534013f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d298423ccb1852 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.811+0000 7f75327fc640 1 -- 172.21.15.78:0/825675360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534013540 con 0x5623e7b279b0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75327fc640 1 -- 172.21.15.78:0/825675360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534011c30 con 0x5623e7b279b0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75327fc640 1 -- 172.21.15.78:0/825675360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753401d9f0 con 0x5623e7b279b0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f7540edf640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 0x7f75280a5920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7534020000 tx=0x7f753401f000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 -- 172.21.15.78:0/825675360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 msgr2=0x5623e7b27d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7534013aa0 tx=0x7f7534013f60 comp rx=0 tx=0).stop 2024-09-16T01:21:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 -- 172.21.15.78:0/825675360 shutdown_connections 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b282d0 0x5623e7b36b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 --2- 172.21.15.78:0/825675360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b279b0 0x5623e7b27d90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 -- 172.21.15.78:0/825675360 >> 172.21.15.78:0/825675360 conn(0x5623e7b24060 msgr2=0x5623e7b24440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 -- 172.21.15.78:0/825675360 shutdown_connections 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.812+0000 7f75456d9740 1 -- 172.21.15.78:0/825675360 wait complete. 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 -- start start 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 Processor -- start 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b34040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b282d0 0x5623e7b34580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5623e7b39c90 con 0x5623e7b370c0 2024-09-16T01:21:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5623e7a15290 con 0x5623e7b279b0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5623e7b39e10 con 0x5623e7b282d0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f75456d9740 10 client.0 _ll_register_callbacks cb 0x5623e7a46140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56572/0 (socket says 172.21.15.78:56572) 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 -- 172.21.15.78:0/1429568549 learned_addr learned my addr 172.21.15.78:0/1429568549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7533fff640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b282d0 0x5623e7b34580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f753bfff640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b34040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 -- 172.21.15.78:0/1429568549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b282d0 msgr2=0x5623e7b34580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b282d0 0x5623e7b34580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 -- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 msgr2=0x5623e7b34040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b34040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.813+0000 7f7540edf640 1 -- 172.21.15.78:0/1429568549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e779f210 con 0x5623e7b370c0 2024-09-16T01:21:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f7540edf640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f753401d5b0 tx=0x7f75340099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e935b00a629e03e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753401d790 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f753401d930 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7534011c30 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f753403c050 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 10 client.16641 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5623e786b620 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534011c30 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7534037030 con 0x5623e7b370c0 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 10 client.16641 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 20 client.16641 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7b47040 0x5623e7b49430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 4 client.16641 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.814+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623e7a99a20 con 0x5623e7b47040 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.815+0000 7f753bfff640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7b47040 0x5623e7b49430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.815+0000 7f751b7fe640 10 client.16641 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.815+0000 7f753bfff640 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7b47040 0x5623e7b49430 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd24a5ffed8ccb34 server_cookie=b9717341a8c8e27d in_seq=0 out_seq=0 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f751b7fe640 1 -- 172.21.15.78:0/1429568549 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5623e7a99a20 con 0x5623e7b47040 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f751b7fe640 10 client.16641 handle_command_reply: tid=0 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 client.16641 shutdown 2024-09-16T01:21:55.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 msgr2=0x5623e7b34ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f753401d5b0 tx=0x7f75340099a0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 shutdown_connections 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7b47040 0x5623e7b49430 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b370c0 0x5623e7b34ac0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f751b7fe640 0 client.16641 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b282d0 0x5623e7b34580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 --2- 172.21.15.78:0/1429568549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 >> 172.21.15.78:0/1429568549 conn(0x5623e7b24060 msgr2=0x5623e7b29cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 shutdown_connections 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 1 -- 172.21.15.78:0/1429568549 wait complete. 2024-09-16T01:21:55.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.816+0000 7f75456d9740 20 client.16641 trim_cache size 0 max 16384 2024-09-16T01:21:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 Processor -- start 2024-09-16T01:21:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 -- start start 2024-09-16T01:21:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b4eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b31280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1af6c0 con 0x5623e7b282d0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1acf90 con 0x5623e7b279b0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f75456d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1aae00 con 0x5623e7b370c0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f7533fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f7533fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56582/0 (socket says 172.21.15.78:56582) 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.830+0000 7f7533fff640 1 -- 172.21.15.78:0/1670299666 learned_addr learned my addr 172.21.15.78:0/1670299666 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f753bfff640 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b4eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7540edf640 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b31280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 -- 172.21.15.78:0/1670299666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 msgr2=0x5623e7b31280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b31280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 -- 172.21.15.78:0/1670299666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 msgr2=0x5623e7b4eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b4eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 -- 172.21.15.78:0/1670299666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e7b2ee90 con 0x5623e7b282d0 2024-09-16T01:21:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7533fff640 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7524005d00 tx=0x7f7524011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c387c524ff2e86b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7531ffb640 1 -- 172.21.15.78:0/1670299666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752401b3f0 con 0x5623e7b282d0 2024-09-16T01:21:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7531ffb640 1 -- 172.21.15.78:0/1670299666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7524020070 con 0x5623e7b282d0 2024-09-16T01:21:55.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.831+0000 7f7531ffb640 1 -- 172.21.15.78:0/1670299666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524010a10 con 0x5623e7b282d0 2024-09-16T01:21:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- 172.21.15.78:0/1670299666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 msgr2=0x5623e7b4f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7524005d00 tx=0x7f7524011810 comp rx=0 tx=0).stop 2024-09-16T01:21:55.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- 172.21.15.78:0/1670299666 shutdown_connections 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b370c0 0x5623e7b31280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b4f000 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- 172.21.15.78:0/1670299666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b279b0 0x5623e7b4eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- 172.21.15.78:0/1670299666 >> 172.21.15.78:0/1670299666 conn(0x5623e7b24060 msgr2=0x5623e7b360c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- 172.21.15.78:0/1670299666 shutdown_connections 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- 172.21.15.78:0/1670299666 wait complete. 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- start start 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 Processor -- start 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 0x5623e7b3b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 0x5623e7b3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1ab450 con 0x5623e7b282d0 2024-09-16T01:21:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1ac8c0 con 0x5623e7b370c0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1abb20 con 0x5623e7b279b0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f75456d9740 10 client.0 _ll_register_callbacks cb 0x7f752400b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f7533fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f7533fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56594/0 (socket says 172.21.15.78:56594) 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.832+0000 7f7533fff640 1 -- 172.21.15.78:0/1375690732 learned_addr learned my addr 172.21.15.78:0/1375690732 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7540edf640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 0x5623e7b3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f753bfff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 0x5623e7b3b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 -- 172.21.15.78:0/1375690732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 msgr2=0x5623e7b3b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 0x5623e7b3b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 -- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 msgr2=0x5623e7b3baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 0x5623e7b3baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 -- 172.21.15.78:0/1375690732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e7bb0c90 con 0x5623e7b282d0 2024-09-16T01:21:55.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f753bfff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 0x5623e7b3b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7540edf640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 0x5623e7b3baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f7533fff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7524005d00 tx=0x7f7524010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f24d0fe9798b160 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524029070 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f7524005450 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7524020070 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7524036050 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f75456d9740 10 client.16647 fetch_fsmap learned FSMap version 31 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5623e7bb1f80 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7524020070 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.833+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7524004bb0 con 0x5623e7b282d0 2024-09-16T01:21:55.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 10 client.16647 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 20 client.16647 populate_metadata read hostname 'smithi078' 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7bb26a0 0x5623e7bb4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 4 client.16647 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623e7a93d70 con 0x5623e7bb26a0 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f753bfff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7bb26a0 0x5623e7bb4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f753bfff640 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7bb26a0 0x5623e7bb4ac0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bad2196783be1b3d server_cookie=9dc6ef6468040def in_seq=0 out_seq=0 2024-09-16T01:21:55.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.834+0000 7f751b7fe640 10 client.16647 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f751b7fe640 1 -- 172.21.15.78:0/1375690732 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623e7a93d70 con 0x5623e7bb26a0 2024-09-16T01:21:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f751b7fe640 10 client.16647 handle_command_reply: tid=0 2024-09-16T01:21:55.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 client.16647 shutdown 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 msgr2=0x5623e7b3b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7524005d00 tx=0x7f7524010430 comp rx=0 tx=0).stop 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 shutdown_connections 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5623e7bb26a0 0x5623e7bb4ac0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5623e7b370c0 0x5623e7b3baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f751b7fe640 0 client.16647 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623e7b282d0 0x5623e7b3b560 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 --2- 172.21.15.78:0/1375690732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5623e7b279b0 0x5623e7b3b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 >> 172.21.15.78:0/1375690732 conn(0x5623e7b24060 msgr2=0x5623e7b229a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 shutdown_connections 2024-09-16T01:21:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 1 -- 172.21.15.78:0/1375690732 wait complete. 2024-09-16T01:21:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f75456d9740 20 client.16647 trim_cache size 0 max 16384 2024-09-16T01:21:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:21:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:21:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:21:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:21:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 msgr2=0x7f75280a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 0x7f75280a5920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7534020000 tx=0x7f753401f000 comp rx=0 tx=0).stop 2024-09-16T01:21:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 msgr2=0x7f753c198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f752c006010 tx=0x7f752c0042f0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.842+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 shutdown_connections 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f75280a3460 0x7f75280a5920 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f753c10c0d0 0x7f753c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f753c103620 0x7f753c198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 --2- 172.21.15.78:0/1082750104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f753c102d00 0x7f753c198340 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 >> 172.21.15.78:0/1082750104 conn(0x7f753c0fc6f0 msgr2=0x7f753c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 shutdown_connections 2024-09-16T01:21:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:21:55.843+0000 7f7542942640 1 -- 172.21.15.78:0/1082750104 wait complete. 2024-09-16T01:21:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 640.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:21:57.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:57 smithi078 ceph-mon[30937]: pgmap v347: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 727 KiB/s wr, 63 op/s 2024-09-16T01:21:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:57 smithi193 ceph-mon[35701]: pgmap v347: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 727 KiB/s wr, 63 op/s 2024-09-16T01:21:57.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:57 smithi148 ceph-mon[35030]: pgmap v347: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 727 KiB/s wr, 63 op/s 2024-09-16T01:21:57.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:21:57.964 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:21:58.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 -- 172.21.15.78:0/2974523836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc41036e0 msgr2=0x7f6fc4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 --2- 172.21.15.78:0/2974523836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc41036e0 0x7f6fc4107b90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009930 tx=0x7f6fac02a420 comp rx=0 tx=0).stop 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 -- 172.21.15.78:0/2974523836 shutdown_connections 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 --2- 172.21.15.78:0/2974523836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41082a0 0x7f6fc410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 --2- 172.21.15.78:0/2974523836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc41036e0 0x7f6fc4107b90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 --2- 172.21.15.78:0/2974523836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc4102d30 0x7f6fc4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 -- 172.21.15.78:0/2974523836 >> 172.21.15.78:0/2974523836 conn(0x7f6fc40fc7d0 msgr2=0x7f6fc40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 -- 172.21.15.78:0/2974523836 shutdown_connections 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.496+0000 7f6fcb131640 1 -- 172.21.15.78:0/2974523836 wait complete. 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 Processor -- start 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 -- start start 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41036e0 0x7f6fc4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc41082a0 0x7f6fc419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc41120a0 con 0x7f6fc4102d30 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4111f20 con 0x7f6fc41036e0 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fcb131640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4112220 con 0x7f6fc41082a0 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fc8ecd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fc8ecd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56614/0 (socket says 172.21.15.78:56614) 2024-09-16T01:21:58.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.497+0000 7f6fc8ecd640 1 -- 172.21.15.78:0/404476227 learned_addr learned my addr 172.21.15.78:0/404476227 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc96ce640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc41082a0 0x7f6fc419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc3fff640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41036e0 0x7f6fc4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 -- 172.21.15.78:0/404476227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc41082a0 msgr2=0x7f6fc419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc41082a0 0x7f6fc419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 -- 172.21.15.78:0/404476227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41036e0 msgr2=0x7f6fc4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:58.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41036e0 0x7f6fc4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 -- 172.21.15.78:0/404476227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc419d590 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc8ecd640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d770 tx=0x7f6fbc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017070 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc419d880 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc005500 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc004e00 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.498+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc41a5160 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.499+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc4103110 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.500+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6fbc004fa0 con 0x7f6fc4102d30 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.501+0000 7f6fc1ffb640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 0x7f6fb00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.501+0000 7f6fc3fff640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 0x7f6fb00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:21:58.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.501+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f6fbc0d0070 con 0x7f6fc4102d30 2024-09-16T01:21:58.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.501+0000 7f6fc3fff640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 0x7f6fb00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009900 tx=0x7f6fac005e50 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:21:58.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.504+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6fbc09c660 con 0x7f6fc4102d30 2024-09-16T01:21:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.707+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fc410a2b0 con 0x7f6fc4102d30 2024-09-16T01:21:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.707+0000 7f6fc1ffb640 1 -- 172.21.15.78:0/404476227 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f6fbc09bf60 con 0x7f6fc4102d30 2024-09-16T01:21:58.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:21:58.709 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:21:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:21:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 msgr2=0x7f6fb00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 0x7f6fb00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009900 tx=0x7f6fac005e50 comp rx=0 tx=0).stop 2024-09-16T01:21:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 msgr2=0x7f6fc41985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:21:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d770 tx=0x7f6fbc00dc30 comp rx=0 tx=0).stop 2024-09-16T01:21:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 shutdown_connections 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fb00a33b0 0x7f6fb00a5870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fc41082a0 0x7f6fc419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fc41036e0 0x7f6fc4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 --2- 172.21.15.78:0/404476227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fc4102d30 0x7f6fc41985e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 >> 172.21.15.78:0/404476227 conn(0x7f6fc40fc7d0 msgr2=0x7f6fc4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 shutdown_connections 2024-09-16T01:21:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:21:58.710+0000 7f6fcb131640 1 -- 172.21.15.78:0/404476227 wait complete. 2024-09-16T01:21:59.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:59 smithi078 ceph-mon[30937]: pgmap v348: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-09-16T01:21:59.386 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:21:59 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/404476227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:59.410 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id dde09c3c --release --await' 2024-09-16T01:21:59.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id dde09c3c --release --await 2024-09-16T01:21:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:59 smithi193 ceph-mon[35701]: pgmap v348: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-09-16T01:21:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:21:59 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/404476227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:59.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:59 smithi148 ceph-mon[35030]: pgmap v348: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 38 KiB/s wr, 11 op/s 2024-09-16T01:21:59.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:21:59 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/404476227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:21:59.658 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:22:00.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 -- 172.21.15.78:0/221767833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca201008d0 msgr2=0x7fca20100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 --2- 172.21.15.78:0/221767833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca201008d0 0x7fca20100cb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009900 tx=0x7fca1c02a410 comp rx=0 tx=0).stop 2024-09-16T01:22:00.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 -- 172.21.15.78:0/221767833 shutdown_connections 2024-09-16T01:22:00.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 --2- 172.21.15.78:0/221767833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 0x7fca20110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 --2- 172.21.15.78:0/221767833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca20101280 0x7fca2010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 --2- 172.21.15.78:0/221767833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca201008d0 0x7fca20100cb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 -- 172.21.15.78:0/221767833 >> 172.21.15.78:0/221767833 conn(0x7fca200fc7a0 msgr2=0x7fca200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 -- 172.21.15.78:0/221767833 shutdown_connections 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.164+0000 7fca27410640 1 -- 172.21.15.78:0/221767833 wait complete. 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 Processor -- start 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 -- start start 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 0x7fca2019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 0x7fca2019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fca20114200 con 0x7fca20101280 2024-09-16T01:22:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fca20114080 con 0x7fca2010e480 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca27410640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fca20114380 con 0x7fca201008d0 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56628/0 (socket says 172.21.15.78:56628) 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 -- 172.21.15.78:0/752491373 learned_addr learned my addr 172.21.15.78:0/752491373 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca251ac640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 0x7fca2019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 -- 172.21.15.78:0/752491373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 msgr2=0x7fca2019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 0x7fca2019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 -- 172.21.15.78:0/752491373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 msgr2=0x7fca2019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca259ad640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 0x7fca2019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 0x7fca2019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca249ab640 1 -- 172.21.15.78:0/752491373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2019f6f0 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.165+0000 7fca251ac640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 0x7fca2019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca249ab640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b4a0 tx=0x7fca1000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca167fc640 1 -- 172.21.15.78:0/752491373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10014070 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2019f9e0 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca167fc640 1 -- 172.21.15.78:0/752491373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca10010af0 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca167fc640 1 -- 172.21.15.78:0/752491373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10017d10 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.166+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca201a72c0 con 0x7fca20101280 2024-09-16T01:22:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.168+0000 7fca167fc640 1 -- 172.21.15.78:0/752491373 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fca10017070 con 0x7fca20101280 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 Processor -- start 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 -- start start 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca167fc640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 0x7fca0c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42ad440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c9b4212620 con 0x55c9b42b8240 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c9b41b9e50 con 0x55c9b42ad060 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca297e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9b420e5e0 con 0x55c9b42ad980 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca259ad640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 0x7fca0c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca167fc640 1 -- 172.21.15.78:0/752491373 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fca100cf4e0 con 0x7fca20101280 2024-09-16T01:22:00.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca259ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca259ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56636/0 (socket says 172.21.15.78:56636) 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca259ad640 1 -- 172.21.15.78:0/2765102678 learned_addr learned my addr 172.21.15.78:0/2765102678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca251ac640 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.169+0000 7fca249ab640 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 -- 172.21.15.78:0/2765102678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 msgr2=0x55c9b42b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 -- 172.21.15.78:0/2765102678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 msgr2=0x55c9b42ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 -- 172.21.15.78:0/2765102678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9b41aad40 con 0x55c9b42b8240 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 0x7fca0c0a5870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fca18012a60 tx=0x7fca1800ac90 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca259ad640 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fca1800fc60 tx=0x7fca18019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2aa5057cc5b05e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca0b7fe640 1 -- 172.21.15.78:0/2765102678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1800dde0 con 0x55c9b42b8240 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca0b7fe640 1 -- 172.21.15.78:0/2765102678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca18011470 con 0x55c9b42b8240 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.170+0000 7fca0b7fe640 1 -- 172.21.15.78:0/2765102678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1801e620 con 0x55c9b42b8240 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- 172.21.15.78:0/2765102678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 msgr2=0x55c9b42ba630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fca1800fc60 tx=0x7fca18019000 comp rx=0 tx=0).stop 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- 172.21.15.78:0/2765102678 shutdown_connections 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42ba630 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 --2- 172.21.15.78:0/2765102678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- 172.21.15.78:0/2765102678 >> 172.21.15.78:0/2765102678 conn(0x55c9b42a95f0 msgr2=0x55c9b42a99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- 172.21.15.78:0/2765102678 shutdown_connections 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- 172.21.15.78:0/2765102678 wait complete. 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 -- start start 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.171+0000 7fca297e9740 1 Processor -- start 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c9b42bae10 con 0x55c9b42b8240 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c9b419af10 con 0x55c9b42ad060 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9b42baf90 con 0x55c9b42ad980 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 10 client.0 _ll_register_callbacks cb 0x55c9b41bedb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:00.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56642/0 (socket says 172.21.15.78:56642) 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 -- 172.21.15.78:0/3558193588 learned_addr learned my addr 172.21.15.78:0/3558193588 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca251ac640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca249ab640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 -- 172.21.15.78:0/3558193588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 msgr2=0x55c9b42b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 -- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 msgr2=0x55c9b42b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 -- 172.21.15.78:0/3558193588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9b3f21580 con 0x55c9b42b8240 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca259ad640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fca18011e30 tx=0x7fca1800a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c3f2ca19e55ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18034060 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fca180121b0 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.172+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca18011470 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fca1803b050 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 10 client.16665 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c9b3feb960 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca18011470 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fca18011820 con 0x55c9b42b8240 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 10 client.16665 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:00.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 10 client.16665 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 20 client.16665 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b42c7d00 0x55c9b42ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 4 client.16665 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9b421fb90 con 0x55c9b42c7d00 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.173+0000 7fca251ac640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b42c7d00 0x55c9b42ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.174+0000 7fca08ff9640 10 client.16665 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.174+0000 7fca251ac640 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b42c7d00 0x55c9b42ca0f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=771ea8449ef973bb server_cookie=410a6a2da15942f7 in_seq=0 out_seq=0 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca08ff9640 1 -- 172.21.15.78:0/3558193588 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9b421fb90 con 0x55c9b42c7d00 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca08ff9640 10 client.16665 handle_command_reply: tid=0 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 client.16665 shutdown 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 msgr2=0x55c9b42b5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fca18011e30 tx=0x7fca1800a6c0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 shutdown_connections 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b42c7d00 0x55c9b42ca0f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42b8240 0x55c9b42b5bc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad980 0x55c9b42b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 --2- 172.21.15.78:0/3558193588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad060 0x55c9b42b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca08ff9640 0 client.16665 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 >> 172.21.15.78:0/3558193588 conn(0x55c9b42a95f0 msgr2=0x55c9b42af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 shutdown_connections 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 1 -- 172.21.15.78:0/3558193588 wait complete. 2024-09-16T01:22:00.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.175+0000 7fca297e9740 20 client.16665 trim_cache size 0 max 16384 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 Processor -- start 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 -- start start 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad980 0x55c9b42cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 0x55c9b42cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fca201b3c50 con 0x55c9b42ad060 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.192+0000 7fca297e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fca201b1520 con 0x55c9b42ad980 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca297e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fca201af390 con 0x55c9b42b8240 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56652/0 (socket says 172.21.15.78:56652) 2024-09-16T01:22:00.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 -- 172.21.15.78:0/389516536 learned_addr learned my addr 172.21.15.78:0/389516536 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca259ad640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 0x55c9b42cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca249ab640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad980 0x55c9b42cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 -- 172.21.15.78:0/389516536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 msgr2=0x55c9b42cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 0x55c9b42cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 -- 172.21.15.78:0/389516536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad980 msgr2=0x55c9b42cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad980 0x55c9b42cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 -- 172.21.15.78:0/389516536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9b42b36b0 con 0x55c9b42ad060 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca259ad640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 0x55c9b42cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca251ac640 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fca1c032fd0 tx=0x7fca1c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae1f02a43384abc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca0affd640 1 -- 172.21.15.78:0/389516536 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c047020 con 0x55c9b42ad060 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca0affd640 1 -- 172.21.15.78:0/389516536 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c032990 con 0x55c9b42ad060 2024-09-16T01:22:00.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.193+0000 7fca0affd640 1 -- 172.21.15.78:0/389516536 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c035040 con 0x55c9b42ad060 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- 172.21.15.78:0/389516536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 msgr2=0x55c9b42a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fca1c032fd0 tx=0x7fca1c005a20 comp rx=0 tx=0).stop 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- 172.21.15.78:0/389516536 shutdown_connections 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42b8240 0x55c9b42cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42ad980 0x55c9b42cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 --2- 172.21.15.78:0/389516536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad060 0x55c9b42a8640 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- 172.21.15.78:0/389516536 >> 172.21.15.78:0/389516536 conn(0x55c9b42a95f0 msgr2=0x55c9b42a7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- 172.21.15.78:0/389516536 shutdown_connections 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- 172.21.15.78:0/389516536 wait complete. 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 -- start start 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 Processor -- start 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.194+0000 7fca297e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad060 0x55c9b42c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42b8240 0x55c9b42bd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fca201af9e0 con 0x55c9b42ad980 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fca201b0e50 con 0x55c9b42b8240 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fca201b00b0 con 0x55c9b42ad060 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 10 client.0 _ll_register_callbacks cb 0x7fca1800a5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56656/0 (socket says 172.21.15.78:56656) 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 -- 172.21.15.78:0/3044610616 learned_addr learned my addr 172.21.15.78:0/3044610616 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca251ac640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad060 0x55c9b42c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca259ad640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42b8240 0x55c9b42bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 -- 172.21.15.78:0/3044610616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad060 msgr2=0x55c9b42c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad060 0x55c9b42c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 -- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42b8240 msgr2=0x55c9b42bd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42b8240 0x55c9b42bd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 -- 172.21.15.78:0/3044610616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9b42bd9e0 con 0x55c9b42ad980 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca249ab640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca100cc950 tx=0x7fca1009b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf66ab315d4aa254 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:00.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1001e040 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fca100cf1b0 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca1c032990 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fca100a40f0 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 10 client.16671 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.195+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c9b4333110 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1c032990 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fca100b3070 con 0x55c9b42ad980 2024-09-16T01:22:00.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 10 client.16671 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 10 client.16671 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 20 client.16671 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b4333840 0x55c9b4335c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 4 client.16671 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dde09c3c", "release": true, "await": true}] 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dde09c3c", "release": true, "await": true}) -- 0x55c9b421a0a0 con 0x55c9b4333840 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca251ac640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b4333840 0x55c9b4335c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca08ff9640 10 client.16671 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:00.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.196+0000 7fca251ac640 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b4333840 0x55c9b4335c60 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31c9c0f72210e2c4 server_cookie=8a45909b111d772e in_seq=0 out_seq=0 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca08ff9640 1 -- 172.21.15.78:0/3044610616 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c9b421a0a0 con 0x55c9b4333840 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca08ff9640 10 client.16671 handle_command_reply: tid=0 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 client.16671 shutdown 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 msgr2=0x55c9b42c0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca100cc950 tx=0x7fca1009b480 comp rx=0 tx=0).stop 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 shutdown_connections 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9b4333840 0x55c9b4335c60 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9b42b8240 0x55c9b42bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9b42ad980 0x55c9b42c0950 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 --2- 172.21.15.78:0/3044610616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9b42ad060 0x55c9b42c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 >> 172.21.15.78:0/3044610616 conn(0x55c9b42a95f0 msgr2=0x55c9b42af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca08ff9640 0 client.16671 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 shutdown_connections 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 1 -- 172.21.15.78:0/3044610616 wait complete. 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.277+0000 7fca297e9740 20 client.16671 trim_cache size 0 max 16384 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:22:00.280 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 40, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "dde09c3c": { 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 40, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:22:00.281 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 141.8, 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:00.282 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 msgr2=0x7fca0c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 0x7fca0c0a5870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fca18012a60 tx=0x7fca1800ac90 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 msgr2=0x7fca2019ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b4a0 tx=0x7fca1000b960 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 shutdown_connections 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fca0c0a33b0 0x7fca0c0a5870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca2010e480 0x7fca2019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca20101280 0x7fca2019ac80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 --2- 172.21.15.78:0/752491373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca201008d0 0x7fca2019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:00.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.278+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 >> 172.21.15.78:0/752491373 conn(0x7fca200fc7a0 msgr2=0x7fca200feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:00.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.284+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 shutdown_connections 2024-09-16T01:22:00.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:00.284+0000 7fca27410640 1 -- 172.21.15.78:0/752491373 wait complete. 2024-09-16T01:22:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 645.0 s, 170.0 tps, lat 2467.153 ms stddev 6598.390 2024-09-16T01:22:01.036 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dde09c3c', total seconds elapsed: 25.2 2024-09-16T01:22:01.037 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-16T01:22:01.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:22:01.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:01 smithi078 ceph-mon[30937]: pgmap v349: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-09-16T01:22:01.467 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:22:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:01 smithi193 ceph-mon[35701]: pgmap v349: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-09-16T01:22:01.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:01 smithi148 ceph-mon[35030]: pgmap v349: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 10 op/s 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.194+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/3932386122 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e4109450 msgr2=0x7fe3e4110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.194+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/3932386122 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e4109450 0x7fe3e4110250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009ca0 tx=0x7fe3dc02a8f0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/3932386122 shutdown_connections 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/3932386122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3e411b140 0x7fe3e411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/3932386122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3e4109830 0x7fe3e4109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/3932386122 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e4109450 0x7fe3e4110250 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/3932386122 >> 172.21.15.78:0/3932386122 conn(0x7fe3e406e140 msgr2=0x7fe3e406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.195+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/3932386122 shutdown_connections 2024-09-16T01:22:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.197+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/3932386122 wait complete. 2024-09-16T01:22:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.198+0000 7fe3e9f1a640 1 Processor -- start 2024-09-16T01:22:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 -- start start 2024-09-16T01:22:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3e4109450 0x7fe3e411a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3e4109830 0x7fe3e4113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e4120e90 con 0x7fe3e4109830 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e4120d10 con 0x7fe3e411b140 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e9f1a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e4121010 con 0x7fe3e4109450 2024-09-16T01:22:02.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47662/0 (socket says 172.21.15.78:47662) 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.199+0000 7fe3e3fff640 1 -- 172.21.15.78:0/280488702 learned_addr learned my addr 172.21.15.78:0/280488702 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.200+0000 7fe3e3fff640 1 -- 172.21.15.78:0/280488702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3e4109450 msgr2=0x7fe3e411a800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.200+0000 7fe3e3fff640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3e4109450 0x7fe3e411a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.200+0000 7fe3e3fff640 1 -- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3e4109830 msgr2=0x7fe3e4113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.200+0000 7fe3e3fff640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3e4109830 0x7fe3e4113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.200+0000 7fe3e3fff640 1 -- 172.21.15.78:0/280488702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e4114470 con 0x7fe3e411b140 2024-09-16T01:22:02.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.201+0000 7fe3e3fff640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c410 tx=0x7fe3d8001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.203+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8004020 con 0x7fe3e411b140 2024-09-16T01:22:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.203+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e4118330 con 0x7fe3e411b140 2024-09-16T01:22:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.203+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e4118870 con 0x7fe3e411b140 2024-09-16T01:22:02.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.205+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3d80041c0 con 0x7fe3e411b140 2024-09-16T01:22:02.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.205+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8011a10 con 0x7fe3e411b140 2024-09-16T01:22:02.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.206+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d00052c0 con 0x7fe3e411b140 2024-09-16T01:22:02.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.208+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fe3d8019020 con 0x7fe3e411b140 2024-09-16T01:22:02.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.209+0000 7fe3e0ff9640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 0x7fe3cc0a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:02.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.209+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7fe3d80cf230 con 0x7fe3e411b140 2024-09-16T01:22:02.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.213+0000 7fe3e37fe640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 0x7fe3cc0a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:02.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.214+0000 7fe3e37fe640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 0x7fe3cc0a5a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc004af0 tx=0x7fe3dc000f40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:02.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.220+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe3d809b9e0 con 0x7fe3e411b140 2024-09-16T01:22:02.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.615+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3d00055b0 con 0x7fe3e411b140 2024-09-16T01:22:02.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.620+0000 7fe3e0ff9640 1 -- 172.21.15.78:0/280488702 <== mon.2 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fe3d809b2e0 con 0x7fe3e411b140 2024-09-16T01:22:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:22:02.622 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:22:02.622 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:22:02.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 msgr2=0x7fe3cc0a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:02.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 0x7fe3cc0a5a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc004af0 tx=0x7fe3dc000f40 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 msgr2=0x7fe3e4113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c410 tx=0x7fe3d8001e70 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 shutdown_connections 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fe3cc0a35c0 0x7fe3cc0a5a80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3e411b140 0x7fe3e4113dd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3e4109830 0x7fe3e4113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 --2- 172.21.15.78:0/280488702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3e4109450 0x7fe3e411a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.625+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 >> 172.21.15.78:0/280488702 conn(0x7fe3e406e140 msgr2=0x7fe3e410ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:02.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.626+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 shutdown_connections 2024-09-16T01:22:02.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:02.629+0000 7fe3e9f1a640 1 -- 172.21.15.78:0/280488702 wait complete. 2024-09-16T01:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:03 smithi078 ceph-mon[30937]: pgmap v350: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-16T01:22:03.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:03 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/280488702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:03 smithi193 ceph-mon[35701]: pgmap v350: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-16T01:22:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:03 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/280488702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:03.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:03 smithi148 ceph-mon[35030]: pgmap v350: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 49 KiB/s wr, 13 op/s 2024-09-16T01:22:03.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:03 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/280488702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:04.737 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-16T01:22:04.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-16T01:22:05.169 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:22:05.492 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:05 smithi078 ceph-mon[30937]: pgmap v351: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-09-16T01:22:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:05 smithi193 ceph-mon[35701]: pgmap v351: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-09-16T01:22:05.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:05 smithi148 ceph-mon[35030]: pgmap v351: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-09-16T01:22:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.896+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3034163115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd010ea80 msgr2=0x7f6fd01154f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:05.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.896+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3034163115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd010ea80 0x7f6fd01154f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009be0 tx=0x7f6fc802c920 comp rx=0 tx=0).stop 2024-09-16T01:22:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.899+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3034163115 shutdown_connections 2024-09-16T01:22:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.899+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3034163115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd010ea80 0x7f6fd01154f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.899+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3034163115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd0109e00 0x7f6fd010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.899+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3034163115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd0109450 0x7f6fd0109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.899+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3034163115 >> 172.21.15.78:0/3034163115 conn(0x7f6fd006e2d0 msgr2=0x7f6fd006e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.900+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3034163115 shutdown_connections 2024-09-16T01:22:05.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.903+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3034163115 wait complete. 2024-09-16T01:22:05.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.904+0000 7f6fd80ed640 1 Processor -- start 2024-09-16T01:22:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 -- start start 2024-09-16T01:22:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd0109450 0x7f6fd0113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd010ea80 0x7f6fd01149b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd0117bc0 con 0x7f6fd0109e00 2024-09-16T01:22:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd0117a40 con 0x7f6fd010ea80 2024-09-16T01:22:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.905+0000 7f6fd80ed640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd0117d40 con 0x7f6fd0109450 2024-09-16T01:22:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57998/0 (socket says 172.21.15.78:57998) 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 -- 172.21.15.78:0/3271446833 learned_addr learned my addr 172.21.15.78:0/3271446833 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd010ea80 0x7f6fd01149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 -- 172.21.15.78:0/3271446833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd0109450 msgr2=0x7f6fd0113f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd0109450 0x7f6fd0113f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 -- 172.21.15.78:0/3271446833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd010ea80 msgr2=0x7f6fd01149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd010ea80 0x7f6fd01149b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 -- 172.21.15.78:0/3271446833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd01bbc50 con 0x7f6fd0109e00 2024-09-16T01:22:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.906+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009800 tx=0x7f6fc0009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.907+0000 7f6fc6ffd640 1 -- 172.21.15.78:0/3271446833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0018020 con 0x7f6fd0109e00 2024-09-16T01:22:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.907+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd01bbf40 con 0x7f6fd0109e00 2024-09-16T01:22:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.907+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd01bc3f0 con 0x7f6fd0109e00 2024-09-16T01:22:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.908+0000 7f6fc6ffd640 1 -- 172.21.15.78:0/3271446833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc000ed90 con 0x7f6fd0109e00 2024-09-16T01:22:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.908+0000 7f6fc6ffd640 1 -- 172.21.15.78:0/3271446833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0013370 con 0x7f6fd0109e00 2024-09-16T01:22:05.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.912+0000 7f6fc6ffd640 1 -- 172.21.15.78:0/3271446833 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6fc001b020 con 0x7f6fd0109e00 2024-09-16T01:22:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.913+0000 7f6fda4c6740 1 Processor -- start 2024-09-16T01:22:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.913+0000 7f6fc6ffd640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 0x7f6fbc0a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.913+0000 7f6fc6ffd640 1 -- 172.21.15.78:0/3271446833 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f6fc00d0ad0 con 0x7f6fd0109e00 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.916+0000 7f6fda4c6740 1 -- start start 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa397850 0x5592fa397c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa398170 0x5592fa3a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5592fa2fc160 con 0x5592fa397850 2024-09-16T01:22:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5592fa2a3720 con 0x5592fa398170 2024-09-16T01:22:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592fa2f8120 con 0x5592fa3a2a30 2024-09-16T01:22:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 0x7f6fbc0a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42566/0 (socket says 172.21.15.78:42566) 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.917+0000 7f6fd668a640 1 -- 172.21.15.78:0/2402439804 learned_addr learned my addr 172.21.15.78:0/2402439804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.918+0000 7f6fd668a640 1 -- 172.21.15.78:0/2402439804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa398170 msgr2=0x5592fa3a24f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.918+0000 7f6fd668a640 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa398170 0x5592fa3a24f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.918+0000 7f6fd668a640 1 -- 172.21.15.78:0/2402439804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa397850 msgr2=0x5592fa397c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.918+0000 7f6fd668a640 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa397850 0x5592fa397c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.918+0000 7f6fd668a640 1 -- 172.21.15.78:0/2402439804 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592fa06dba0 con 0x5592fa3a2a30 2024-09-16T01:22:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.919+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 0x7f6fbc0a5a80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8031f90 tx=0x7f6fc800e3e0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.919+0000 7f6fd668a640 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009b30 tx=0x7f6fc8046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22d440f9ae9b1ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.919+0000 7f6fbbfff640 1 -- 172.21.15.78:0/2402439804 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc804f070 con 0x5592fa3a2a30 2024-09-16T01:22:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.919+0000 7f6fbbfff640 1 -- 172.21.15.78:0/2402439804 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc80329e0 con 0x5592fa3a2a30 2024-09-16T01:22:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.919+0000 7f6fbbfff640 1 -- 172.21.15.78:0/2402439804 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc803c450 con 0x5592fa3a2a30 2024-09-16T01:22:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.922+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2402439804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 msgr2=0x5592fa3a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.922+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009b30 tx=0x7f6fc8046000 comp rx=0 tx=0).stop 2024-09-16T01:22:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.922+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2402439804 shutdown_connections 2024-09-16T01:22:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.922+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a4e20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.923+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa398170 0x5592fa3a24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.923+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2402439804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa397850 0x5592fa397c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.923+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2402439804 >> 172.21.15.78:0/2402439804 conn(0x5592fa393de0 msgr2=0x5592fa3941c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:05.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.929+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2402439804 shutdown_connections 2024-09-16T01:22:05.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.932+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2402439804 wait complete. 2024-09-16T01:22:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.933+0000 7f6fda4c6740 1 -- start start 2024-09-16T01:22:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.933+0000 7f6fda4c6740 1 Processor -- start 2024-09-16T01:22:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 650.0 s, 632.8 tps, lat 154.434 ms stddev 188.801 2024-09-16T01:22:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa39fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa39ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5592fa3a5600 con 0x5592fa398170 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5592fa284900 con 0x5592fa3a2a30 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592fa3a5780 con 0x5592fa3a0490 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.935+0000 7f6fda4c6740 10 client.0 _ll_register_callbacks cb 0x5592fa282780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42568/0 (socket says 172.21.15.78:42568) 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 -- 172.21.15.78:0/3531283105 learned_addr learned my addr 172.21.15.78:0/3531283105 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa39ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 -- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 msgr2=0x5592fa39ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa39ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 -- 172.21.15.78:0/3531283105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 msgr2=0x5592fa39fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa39fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.936+0000 7f6fd668a640 1 -- 172.21.15.78:0/3531283105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592fa2c4c50 con 0x5592fa3a0490 2024-09-16T01:22:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.942+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8032010 tx=0x7f6fc802d0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=239ab526756bb873 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.943+0000 7f6fb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:22:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.947+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8059070 con 0x5592fa3a0490 2024-09-16T01:22:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.947+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f6fc803c6d0 con 0x5592fa3a0490 2024-09-16T01:22:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.948+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc80329e0 con 0x5592fa3a0490 2024-09-16T01:22:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.948+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6fc8062050 con 0x5592fa3a0490 2024-09-16T01:22:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.950+0000 7f6fda4c6740 10 client.26053 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.950+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592fa3a5600 con 0x5592fa3a0490 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc80329e0 con 0x5592fa3a0490 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6fc804b8e0 con 0x5592fa3a0490 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 10 client.26053 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 10 client.26053 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 20 client.26053 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa3b2d40 0x5592fa3b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 4 client.26053 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.951+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592fa309890 con 0x5592fa3b2d40 2024-09-16T01:22:05.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.952+0000 7f6fd5e89640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa3b2d40 0x5592fa3b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.954+0000 7f6fb97fa640 10 client.26053 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.954+0000 7f6fd5e89640 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa3b2d40 0x5592fa3b51c0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a12e9e3ef2766c16 server_cookie=6bea2afb85b0d623 in_seq=0 out_seq=0 2024-09-16T01:22:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.957+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3531283105 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5592fa309890 con 0x5592fa3b2d40 2024-09-16T01:22:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.957+0000 7f6fb97fa640 10 client.26053 handle_command_reply: tid=0 2024-09-16T01:22:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.957+0000 7f6fda4c6740 1 client.26053 shutdown 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 msgr2=0x5592fa3aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8032010 tx=0x7f6fc802d0a0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 shutdown_connections 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa3b2d40 0x5592fa3b51c0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3aeb90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa39ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3531283105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa39fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 >> 172.21.15.78:0/3531283105 conn(0x5592fa393de0 msgr2=0x5592fa399ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.962+0000 7f6fb97fa640 0 client.26053 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.963+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 shutdown_connections 2024-09-16T01:22:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.964+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3531283105 wait complete. 2024-09-16T01:22:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.964+0000 7f6fda4c6740 20 client.26053 trim_cache size 0 max 16384 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 Processor -- start 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 -- start start 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3ae980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3baad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa3bb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb4012990 con 0x5592fa398170 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.993+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb4010260 con 0x5592fa3a2a30 2024-09-16T01:22:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.994+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa3bb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.994+0000 7f6fd668a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa3bb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:37140/0 (socket says 172.21.15.78:37140) 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.994+0000 7f6fd668a640 1 -- 172.21.15.78:0/2680356994 learned_addr learned my addr 172.21.15.78:0/2680356994 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.994+0000 7f6fd5688640 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.994+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb400e0d0 con 0x5592fa3a0490 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.995+0000 7f6fd5688640 1 -- 172.21.15.78:0/2680356994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 msgr2=0x5592fa3bb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.995+0000 7f6fd5688640 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa3bb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.995+0000 7f6fd5688640 1 -- 172.21.15.78:0/2680356994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 msgr2=0x5592fa3ae980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:22:05.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.995+0000 7f6fd5688640 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.995+0000 7f6fd5688640 1 -- 172.21.15.78:0/2680356994 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592fa38f3f0 con 0x5592fa3a0490 2024-09-16T01:22:05.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.996+0000 7f6fd5688640 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3baad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00cddd0 tx=0x7f6fc009c2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d6e32f905f0d8e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:05.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.996+0000 7f6fbb7fe640 1 -- 172.21.15.78:0/2680356994 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00b4020 con 0x5592fa3a0490 2024-09-16T01:22:05.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.996+0000 7f6fbb7fe640 1 -- 172.21.15.78:0/2680356994 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc00a0d10 con 0x5592fa3a0490 2024-09-16T01:22:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.996+0000 7f6fbb7fe640 1 -- 172.21.15.78:0/2680356994 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00a53a0 con 0x5592fa3a0490 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 msgr2=0x5592fa3baad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3baad0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00cddd0 tx=0x7f6fc009c2c0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 shutdown_connections 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a2a30 0x5592fa3bb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a0490 0x5592fa3baad0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/2680356994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.997+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 >> 172.21.15.78:0/2680356994 conn(0x5592fa393de0 msgr2=0x5592fa392ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 shutdown_connections 2024-09-16T01:22:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 -- 172.21.15.78:0/2680356994 wait complete. 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 -- start start 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 Processor -- start 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.998+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a0490 0x5592fa3a6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fda4c6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb400e720 con 0x5592fa398170 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb400fb90 con 0x5592fa3a0490 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fda4c6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb400edf0 con 0x5592fa3a2a30 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fda4c6740 10 client.0 _ll_register_callbacks cb 0x7f6fc8009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58036/0 (socket says 172.21.15.78:58036) 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 -- 172.21.15.78:0/3731085774 learned_addr learned my addr 172.21.15.78:0/3731085774 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a0490 0x5592fa3a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 -- 172.21.15.78:0/3731085774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 msgr2=0x5592fa3a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 -- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a0490 msgr2=0x5592fa3a6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a0490 0x5592fa3a6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd5e89640 1 -- 172.21.15.78:0/3731085774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592fa3a6d00 con 0x5592fa398170 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:05.999+0000 7f6fd668a640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fd5e89640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x5592fa395080 tx=0x7f6fcc00bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b998cbe0d6f751 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc01e060 con 0x5592fa398170 2024-09-16T01:22:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9525+0+0 (secure 0 0 0) 0x7f6fcc004b70 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc00a0d10 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6fcc025050 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fda4c6740 10 client.16680 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.000+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592fa41d7f0 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc00a0d10 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6fcc003ae0 con 0x5592fa398170 2024-09-16T01:22:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 10 client.16680 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 10 client.16680 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 20 client.16680 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa41df90 0x5592fa4203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 4 client.16680 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.001+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x5592fa303be0 con 0x5592fa41df90 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.003+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa41df90 0x5592fa4203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.004+0000 7f6fd5688640 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa41df90 0x5592fa4203b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=679ab2fc1c50635c server_cookie=1b7d940f351118d5 in_seq=0 out_seq=0 2024-09-16T01:22:06.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.004+0000 7f6fb97fa640 10 client.16680 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fb97fa640 1 -- 172.21.15.78:0/3731085774 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5592fa303be0 con 0x5592fa41df90 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fb97fa640 10 client.16680 handle_command_reply: tid=0 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 client.16680 shutdown 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 msgr2=0x5592fa3aaf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x5592fa395080 tx=0x7f6fcc00bba0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 shutdown_connections 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5592fa41df90 0x5592fa4203b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592fa3a2a30 0x5592fa3a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5592fa3a0490 0x5592fa3a6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fb97fa640 0 client.16680 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 --2- 172.21.15.78:0/3731085774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592fa398170 0x5592fa3aaf50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 >> 172.21.15.78:0/3731085774 conn(0x5592fa393de0 msgr2=0x5592fa3992a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 shutdown_connections 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.308+0000 7f6fda4c6740 1 -- 172.21.15.78:0/3731085774 wait complete. 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fda4c6740 20 client.16680 trim_cache size 0 max 16384 2024-09-16T01:22:06.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 42, 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "17ca30df": { 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 42, 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:22:06.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 141.8, 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:22:06.313 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:06.314 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 msgr2=0x7f6fbc0a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 0x7f6fbc0a5a80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8031f90 tx=0x7f6fc800e3e0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 msgr2=0x7f6fd0114470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009800 tx=0x7f6fc0009cc0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 shutdown_connections 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6fbc0a35c0 0x7f6fbc0a5a80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6fd010ea80 0x7f6fd01149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fd0109e00 0x7f6fd0114470 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 --2- 172.21.15.78:0/3271446833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd0109450 0x7f6fd0113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 >> 172.21.15.78:0/3271446833 conn(0x7f6fd006e2d0 msgr2=0x7f6fd0073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.309+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 shutdown_connections 2024-09-16T01:22:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:06.310+0000 7f6fd80ed640 1 -- 172.21.15.78:0/3271446833 wait complete. 2024-09-16T01:22:07.035 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '17ca30df', quiesce took 6.0 seconds. Will release after: 15.850000000000001 2024-09-16T01:22:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:07 smithi078 ceph-mon[30937]: pgmap v352: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-09-16T01:22:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:07 smithi193 ceph-mon[35701]: pgmap v352: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-09-16T01:22:07.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:07 smithi148 ceph-mon[35030]: pgmap v352: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.7 MiB/s wr, 106 op/s 2024-09-16T01:22:08.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 -- 172.21.15.78:0/3572171907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c105ea0 msgr2=0x7f469c106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:08.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 --2- 172.21.15.78:0/3572171907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c105ea0 0x7f469c106310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4694009ca0 tx=0x7f469402a980 comp rx=0 tx=0).stop 2024-09-16T01:22:08.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 -- 172.21.15.78:0/3572171907 shutdown_connections 2024-09-16T01:22:08.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 --2- 172.21.15.78:0/3572171907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c105ea0 0x7f469c106310 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:08.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 --2- 172.21.15.78:0/3572171907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f469c101390 0x7f469c105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:08.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 --2- 172.21.15.78:0/3572171907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f469c100a70 0x7f469c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:08.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.985+0000 7f46a15e4640 1 -- 172.21.15.78:0/3572171907 >> 172.21.15.78:0/3572171907 conn(0x7f469c077990 msgr2=0x7f469c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.988+0000 7f46a15e4640 1 -- 172.21.15.78:0/3572171907 shutdown_connections 2024-09-16T01:22:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.988+0000 7f46a15e4640 1 -- 172.21.15.78:0/3572171907 wait complete. 2024-09-16T01:22:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 Processor -- start 2024-09-16T01:22:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 -- start start 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f469c101390 0x7f469c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f469c105ea0 0x7f469c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1185b0 con 0x7f469c101390 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f469c118430 con 0x7f469c105ea0 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.989+0000 7f46a15e4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f469c118730 con 0x7f469c100a70 2024-09-16T01:22:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.990+0000 7f469affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42598/0 (socket says 172.21.15.78:42598) 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 -- 172.21.15.78:0/935323652 learned_addr learned my addr 172.21.15.78:0/935323652 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 -- 172.21.15.78:0/935323652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f469c105ea0 msgr2=0x7f469c19e860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f469c105ea0 0x7f469c19e860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 -- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f469c101390 msgr2=0x7f469c1a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:08.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f469c101390 0x7f469c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 -- 172.21.15.78:0/935323652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c19efe0 con 0x7f469c100a70 2024-09-16T01:22:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.991+0000 7f469affd640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f468c00b450 tx=0x7f468c00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.998+0000 7f4683fff640 1 -- 172.21.15.78:0/935323652 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c015020 con 0x7f469c100a70 2024-09-16T01:22:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.998+0000 7f4683fff640 1 -- 172.21.15.78:0/935323652 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f468c011d70 con 0x7f469c100a70 2024-09-16T01:22:08.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.998+0000 7f4683fff640 1 -- 172.21.15.78:0/935323652 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c00f340 con 0x7f469c100a70 2024-09-16T01:22:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.999+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c19f2d0 con 0x7f469c100a70 2024-09-16T01:22:09.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:08.999+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c1ab570 con 0x7f469c100a70 2024-09-16T01:22:09.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.002+0000 7f4683fff640 1 -- 172.21.15.78:0/935323652 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f468c018020 con 0x7f469c100a70 2024-09-16T01:22:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.003+0000 7f46a4457740 1 Processor -- start 2024-09-16T01:22:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 -- start start 2024-09-16T01:22:09.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f4683fff640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 0x7f46840a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f94324f2f0 con 0x55f9432f5cb0 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f9431f7270 con 0x55f9432eb410 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f46a4457740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f94324b2b0 con 0x55f9432eaaf0 2024-09-16T01:22:09.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.004+0000 7f469a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:37166/0 (socket says 172.21.15.78:37166) 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 -- 172.21.15.78:0/172063613 learned_addr learned my addr 172.21.15.78:0/172063613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469affd640 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 -- 172.21.15.78:0/172063613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 msgr2=0x55f9432eaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432eaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 -- 172.21.15.78:0/172063613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 msgr2=0x55f9432f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 -- 172.21.15.78:0/172063613 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9431f6340 con 0x55f9432eb410 2024-09-16T01:22:09.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f4683fff640 1 -- 172.21.15.78:0/935323652 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9613+0+0 (secure 0 0 0) 0x7f468c0cfad0 con 0x7f469c100a70 2024-09-16T01:22:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.005+0000 7f469a7fc640 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4690009770 tx=0x7f4690009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c136f0f8f7342802 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:09.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.006+0000 7f469b7fe640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 0x7f46840a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.010+0000 7f46817fa640 1 -- 172.21.15.78:0/172063613 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690015020 con 0x55f9432eb410 2024-09-16T01:22:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.010+0000 7f46817fa640 1 -- 172.21.15.78:0/172063613 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690011420 con 0x55f9432eb410 2024-09-16T01:22:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.010+0000 7f46817fa640 1 -- 172.21.15.78:0/172063613 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46900105c0 con 0x55f9432eb410 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 -- 172.21.15.78:0/172063613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 msgr2=0x55f9432f5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4690009770 tx=0x7f4690009c30 comp rx=0 tx=0).stop 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 -- 172.21.15.78:0/172063613 shutdown_connections 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f5770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 --2- 172.21.15.78:0/172063613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 -- 172.21.15.78:0/172063613 >> 172.21.15.78:0/172063613 conn(0x55f9432e7060 msgr2=0x55f9432e7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:09.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f469b7fe640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 0x7f46840a5b50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4694009a60 tx=0x7f4694005f60 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:09.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.011+0000 7f46a4457740 1 -- 172.21.15.78:0/172063613 shutdown_connections 2024-09-16T01:22:09.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 -- 172.21.15.78:0/172063613 wait complete. 2024-09-16T01:22:09.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 -- start start 2024-09-16T01:22:09.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 Processor -- start 2024-09-16T01:22:09.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f9432f88d0 con 0x55f9432f5cb0 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f9431d8290 con 0x55f9432eb410 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f9432f8a50 con 0x55f9432eaaf0 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.012+0000 7f46a4457740 10 client.0 _ll_register_callbacks cb 0x55f943209140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42604/0 (socket says 172.21.15.78:42604) 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 -- 172.21.15.78:0/2015324516 learned_addr learned my addr 172.21.15.78:0/2015324516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469b7fe640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 -- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 msgr2=0x55f9432f30e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f30e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 -- 172.21.15.78:0/2015324516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 msgr2=0x55f9432f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.013+0000 7f469affd640 1 -- 172.21.15.78:0/2015324516 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f942f62210 con 0x55f9432eaaf0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.014+0000 7f469affd640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f468c09b720 tx=0x7f468c0b4ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb810459b894ae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.014+0000 7f467eff5640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.014+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c09fdd0 con 0x55f9432eaaf0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.014+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9613+0+0 (secure 0 0 0) 0x7f468c0ba320 con 0x55f9432eaaf0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.014+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4690011420 con 0x55f9432eaaf0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.015+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f468c0a4690 con 0x55f9432eaaf0 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.015+0000 7f46a4457740 10 client.26065 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:09.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.015+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f94302e620 con 0x55f9432eaaf0 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.016+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690011420 con 0x55f9432eaaf0 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.016+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f468c0b7e60 con 0x55f9432eaaf0 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 10 client.26065 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 10 client.26065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 20 client.26065 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943305d50 0x55f9433081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 4 client.26065 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f94325ca20 con 0x55f943305d50 2024-09-16T01:22:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f469a7fc640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943305d50 0x55f9433081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.017+0000 7f467eff5640 10 client.26065 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.018+0000 7f469a7fc640 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943305d50 0x55f9433081a0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a153450641b80d server_cookie=9922e3401e621ac5 in_seq=0 out_seq=0 2024-09-16T01:22:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f467eff5640 1 -- 172.21.15.78:0/2015324516 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f94325ca20 con 0x55f943305d50 2024-09-16T01:22:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f467eff5640 10 client.26065 handle_command_reply: tid=0 2024-09-16T01:22:09.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 client.26065 shutdown 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 msgr2=0x55f9432f2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f468c09b720 tx=0x7f468c0b4ec0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 shutdown_connections 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943305d50 0x55f9433081a0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f9432f5cb0 0x55f9432f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 --2- 172.21.15.78:0/2015324516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eaaf0 0x55f9432f2ba0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f468c09b720 tx=0x7f468c0b4ec0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 >> 172.21.15.78:0/2015324516 conn(0x55f9432e7060 msgr2=0x55f9432ecd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f467eff5640 0 client.26065 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.020+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 shutdown_connections 2024-09-16T01:22:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.021+0000 7f46a4457740 1 -- 172.21.15.78:0/2015324516 wait complete. 2024-09-16T01:22:09.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.021+0000 7f46a4457740 20 client.26065 trim_cache size 0 max 16384 2024-09-16T01:22:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.049+0000 7f46a4457740 1 Processor -- start 2024-09-16T01:22:09.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.049+0000 7f46a4457740 1 -- start start 2024-09-16T01:22:09.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:37186/0 (socket says 172.21.15.78:37186) 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469a7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469a7fc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42608/0 (socket says 172.21.15.78:42608) 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469affd640 1 -- 172.21.15.78:0/1564482120 learned_addr learned my addr 172.21.15.78:0/1564482120 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b7ff0 con 0x55f94330dff0 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b58c0 con 0x55f9432eb410 2024-09-16T01:22:09.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f469b7fe640 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.050+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b3730 con 0x55f9432f5cb0 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 -- 172.21.15.78:0/1564482120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 msgr2=0x55f9432f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 -- 172.21.15.78:0/1564482120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 msgr2=0x55f9432efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 -- 172.21.15.78:0/1564482120 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9432eb240 con 0x55f9432f5cb0 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f469a7fc640 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55f9432f41e0 tx=0x7f4690011f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd17be868f5f9b4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:09.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.051+0000 7f4680ff9640 1 -- 172.21.15.78:0/1564482120 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690015040 con 0x55f9432f5cb0 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.052+0000 7f4680ff9640 1 -- 172.21.15.78:0/1564482120 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690028030 con 0x55f9432f5cb0 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.052+0000 7f4680ff9640 1 -- 172.21.15.78:0/1564482120 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46900242d0 con 0x55f9432f5cb0 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 msgr2=0x55f94330dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55f9432f41e0 tx=0x7f4690011f40 comp rx=0 tx=0).stop 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 shutdown_connections 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432f5cb0 0x55f94330dab0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 --2- 172.21.15.78:0/1564482120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432eb410 0x55f9432f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 >> 172.21.15.78:0/1564482120 conn(0x55f9432e7060 msgr2=0x55f9432f4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 shutdown_connections 2024-09-16T01:22:09.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- 172.21.15.78:0/1564482120 wait complete. 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 -- start start 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.053+0000 7f46a4457740 1 Processor -- start 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 0x55f9432fe480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432f5cb0 0x55f9432ef950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b3d80 con 0x55f94330dff0 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b51f0 con 0x55f9432f5cb0 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1b4450 con 0x55f9432eb410 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f46a4457740 10 client.0 _ll_register_callbacks cb 0x7f468c09b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58090/0 (socket says 172.21.15.78:58090) 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 -- 172.21.15.78:0/3806402165 learned_addr learned my addr 172.21.15.78:0/3806402165 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469affd640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 0x55f9432fe480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 -- 172.21.15.78:0/3806402165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 msgr2=0x55f9432fe480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 0x55f9432fe480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 -- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432f5cb0 msgr2=0x55f9432ef950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432f5cb0 0x55f9432ef950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 -- 172.21.15.78:0/3806402165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9432f0670 con 0x55f94330dff0 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469affd640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 0x55f9432fe480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f469b7fe640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4694009ca0 tx=0x7f469402a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85932dc2ecf857ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:09.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.054+0000 7f467eff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694055060 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4690028030 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9613+0+0 (secure 0 0 0) 0x7f4694030cd0 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f469405c050 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f46a4457740 10 client.16689 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.055+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f943370af0 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4690028030 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f469404f070 con 0x55f94330dff0 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 10 client.16689 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:09.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 20 client.16689 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943371230 0x55f943373650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 4 client.16689 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f943256d70 con 0x55f943371230 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.056+0000 7f469affd640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943371230 0x55f943373650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.057+0000 7f467eff5640 10 client.16689 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.057+0000 7f469affd640 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943371230 0x55f943373650 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3946da8fd0539bad server_cookie=a8963dbe1d08cd0a in_seq=0 out_seq=0 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f467eff5640 1 -- 172.21.15.78:0/3806402165 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f943256d70 con 0x55f943371230 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f467eff5640 10 client.16689 handle_command_reply: tid=0 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 client.16689 shutdown 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 msgr2=0x55f9432efe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4694009ca0 tx=0x7f469402a980 comp rx=0 tx=0).stop 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 shutdown_connections 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f943371230 0x55f943373650 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f94330dff0 0x55f9432efe90 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f9432f5cb0 0x55f9432ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 --2- 172.21.15.78:0/3806402165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9432eb410 0x55f9432fe480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f467eff5640 0 client.16689 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.080+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 >> 172.21.15.78:0/3806402165 conn(0x55f9432e7060 msgr2=0x55f9432f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 shutdown_connections 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a4457740 1 -- 172.21.15.78:0/3806402165 wait complete. 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a4457740 20 client.16689 trim_cache size 0 max 16384 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:22:09.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:22:09.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 msgr2=0x7f46840a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 0x7f46840a5b50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4694009a60 tx=0x7f4694005f60 comp rx=0 tx=0).stop 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 msgr2=0x7f469c1a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f468c00b450 tx=0x7f468c00b910 comp rx=0 tx=0).stop 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 shutdown_connections 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46840a3690 0x7f46840a5b50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f469c105ea0 0x7f469c19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f469c101390 0x7f469c1a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 --2- 172.21.15.78:0/935323652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f469c100a70 0x7f469c1a46b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.081+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 >> 172.21.15.78:0/935323652 conn(0x7f469c077990 msgr2=0x7f469c103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.082+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 shutdown_connections 2024-09-16T01:22:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:09.082+0000 7f46a15e4640 1 -- 172.21.15.78:0/935323652 wait complete. 2024-09-16T01:22:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:09 smithi078 ceph-mon[30937]: pgmap v353: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 217 op/s 2024-09-16T01:22:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:09 smithi078 ceph-mon[30937]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T01:22:09.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:09 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:22:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:09 smithi193 ceph-mon[35701]: pgmap v353: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 217 op/s 2024-09-16T01:22:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:09 smithi193 ceph-mon[35701]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T01:22:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:09 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:22:09.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:09 smithi148 ceph-mon[35030]: pgmap v353: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 217 op/s 2024-09-16T01:22:09.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:09 smithi148 ceph-mon[35030]: osdmap e105: 12 total, 12 up, 12 in 2024-09-16T01:22:09.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:09 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: pgmap v355: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 337 op/s 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:22:10.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: pgmap v355: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 337 op/s 2024-09-16T01:22:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:22:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:22:10.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: pgmap v355: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.7 MiB/s wr, 337 op/s 2024-09-16T01:22:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:22:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:22:10.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:22:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 655.0 s, 17.4 tps, lat 194.300 ms stddev 182.417 2024-09-16T01:22:12.779 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:12 smithi078 ceph-mon[30937]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 375 op/s 2024-09-16T01:22:12.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:12 smithi193 ceph-mon[35701]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 375 op/s 2024-09-16T01:22:12.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:12 smithi148 ceph-mon[35030]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 375 op/s 2024-09-16T01:22:14.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:14 smithi078 ceph-mon[30937]: pgmap v357: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.8 MiB/s wr, 361 op/s 2024-09-16T01:22:14.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:14 smithi193 ceph-mon[35701]: pgmap v357: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.8 MiB/s wr, 361 op/s 2024-09-16T01:22:14.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:14 smithi148 ceph-mon[35030]: pgmap v357: 129 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 123 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.8 MiB/s wr, 361 op/s 2024-09-16T01:22:15.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 660.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:22:16.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:16 smithi148 ceph-mon[35030]: pgmap v358: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 3.4 MiB/s wr, 290 op/s 2024-09-16T01:22:16.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:16 smithi148 ceph-mon[35030]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T01:22:17.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:16 smithi078 ceph-mon[30937]: pgmap v358: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 3.4 MiB/s wr, 290 op/s 2024-09-16T01:22:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:16 smithi078 ceph-mon[30937]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T01:22:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:16 smithi193 ceph-mon[35701]: pgmap v358: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 3.4 MiB/s wr, 290 op/s 2024-09-16T01:22:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:16 smithi193 ceph-mon[35701]: osdmap e106: 12 total, 12 up, 12 in 2024-09-16T01:22:18.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:18 smithi148 ceph-mon[35030]: pgmap v360: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-09-16T01:22:19.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:18 smithi078 ceph-mon[30937]: pgmap v360: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-09-16T01:22:19.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:18 smithi193 ceph-mon[35701]: pgmap v360: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-09-16T01:22:20.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:20 smithi148 ceph-mon[35030]: pgmap v361: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 757 KiB/s wr, 77 op/s 2024-09-16T01:22:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 665.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:22:21.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:20 smithi078 ceph-mon[30937]: pgmap v361: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 757 KiB/s wr, 77 op/s 2024-09-16T01:22:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:20 smithi193 ceph-mon[35701]: pgmap v361: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 757 KiB/s wr, 77 op/s 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 -- 172.21.15.78:0/2618483619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904102c80 msgr2=0x7fb904103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 --2- 172.21.15.78:0/2618483619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904102c80 0x7fb904103060 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009900 tx=0x7fb8f402a480 comp rx=0 tx=0).stop 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 -- 172.21.15.78:0/2618483619 shutdown_connections 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 --2- 172.21.15.78:0/2618483619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb90410c0d0 0x7fb90410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 --2- 172.21.15.78:0/2618483619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb90410ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 --2- 172.21.15.78:0/2618483619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb904102c80 0x7fb904103060 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 -- 172.21.15.78:0/2618483619 >> 172.21.15.78:0/2618483619 conn(0x7fb904077990 msgr2=0x7fb9040fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 -- 172.21.15.78:0/2618483619 shutdown_connections 2024-09-16T01:22:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.180+0000 7fb90a677640 1 -- 172.21.15.78:0/2618483619 wait complete. 2024-09-16T01:22:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 Processor -- start 2024-09-16T01:22:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 -- start start 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 0x7fb9041008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb904100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb904111eb0 con 0x7fb90410c0d0 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb904111d30 con 0x7fb9041035a0 2024-09-16T01:22:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb90a677640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb904112030 con 0x7fb904102c80 2024-09-16T01:22:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb908c14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb908c14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58490/0 (socket says 172.21.15.78:58490) 2024-09-16T01:22:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb908c14640 1 -- 172.21.15.78:0/2240275223 learned_addr learned my addr 172.21.15.78:0/2240275223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb903fff640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 0x7fb9041008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.181+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb904100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 -- 172.21.15.78:0/2240275223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 msgr2=0x7fb9041008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 0x7fb9041008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 -- 172.21.15.78:0/2240275223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 msgr2=0x7fb904100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb904100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 -- 172.21.15.78:0/2240275223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9041a6880 con 0x7fb90410c0d0 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb904100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb903fff640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 0x7fb9041008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb908c14640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00cad0 tx=0x7fb8fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb901ffb640 1 -- 172.21.15.78:0/2240275223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc00ae20 con 0x7fb90410c0d0 2024-09-16T01:22:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb901ffb640 1 -- 172.21.15.78:0/2240275223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8fc005020 con 0x7fb90410c0d0 2024-09-16T01:22:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9041a6b70 con 0x7fb90410c0d0 2024-09-16T01:22:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb901ffb640 1 -- 172.21.15.78:0/2240275223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc00f6f0 con 0x7fb90410c0d0 2024-09-16T01:22:22.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.182+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9041a70c0 con 0x7fb90410c0d0 2024-09-16T01:22:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.184+0000 7fb901ffb640 1 -- 172.21.15.78:0/2240275223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fb8fc0051d0 con 0x7fb90410c0d0 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 Processor -- start 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 -- start start 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c4ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c5b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f567bb22f0 con 0x55f567c4e410 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f567b5a270 con 0x55f567c58cb0 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb90d412740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f567bae2b0 con 0x55f567c4daf0 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58506/0 (socket says 172.21.15.78:58506) 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 -- 172.21.15.78:0/504377890 learned_addr learned my addr 172.21.15.78:0/504377890 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb901ffb640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 0x7fb8f00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c4ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c5b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb908c14640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 0x7fb8f00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 -- 172.21.15.78:0/504377890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 msgr2=0x55f567c4ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c4ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 -- 172.21.15.78:0/504377890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 msgr2=0x55f567c5b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c5b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb903fff640 1 -- 172.21.15.78:0/504377890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f567b59340 con 0x55f567c4e410 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c5b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.185+0000 7fb901ffb640 1 -- 172.21.15.78:0/2240275223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fb8fc0d0160 con 0x7fb90410c0d0 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb903fff640 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4033000 tx=0x7fb8f4032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad8718444aa6aa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb908c14640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 0x7fb8f00a59a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc09bfe0 tx=0x7fb8fc085040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/504377890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f403f070 con 0x55f567c4e410 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/504377890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f40317b0 con 0x55f567c4e410 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb8fa7fc640 1 -- 172.21.15.78:0/504377890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4035070 con 0x55f567c4e410 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 -- 172.21.15.78:0/504377890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 msgr2=0x55f567c58770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4033000 tx=0x7fb8f4032d60 comp rx=0 tx=0).stop 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 -- 172.21.15.78:0/504377890 shutdown_connections 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c5b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4e410 0x55f567c58770 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 --2- 172.21.15.78:0/504377890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c4ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.186+0000 7fb90d412740 1 -- 172.21.15.78:0/504377890 >> 172.21.15.78:0/504377890 conn(0x55f567c4a060 msgr2=0x55f567c4a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- 172.21.15.78:0/504377890 shutdown_connections 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- 172.21.15.78:0/504377890 wait complete. 2024-09-16T01:22:22.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- start start 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 Processor -- start 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4daf0 0x55f567c55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f567c5b880 con 0x55f567c58cb0 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f567b3b290 con 0x55f567c4daf0 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f567c5ba00 con 0x55f567c4e410 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.187+0000 7fb90d412740 10 client.0 _ll_register_callbacks cb 0x55f567b6c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58520/0 (socket says 172.21.15.78:58520) 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 -- 172.21.15.78:0/651706349 learned_addr learned my addr 172.21.15.78:0/651706349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:22.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4daf0 0x55f567c55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb903fff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 -- 172.21.15.78:0/651706349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 msgr2=0x55f567c560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 -- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4daf0 msgr2=0x55f567c55b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4daf0 0x55f567c55b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 -- 172.21.15.78:0/651706349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5678c5210 con 0x55f567c58cb0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec013de0 tx=0x7fb8ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59a2858e13902ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec004030 con 0x55f567c58cb0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fb8ec0041d0 con 0x55f567c58cb0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8f40317b0 con 0x55f567c58cb0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8ec029050 con 0x55f567c58cb0 2024-09-16T01:22:22.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb90d412740 10 client.16701 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f567991620 con 0x55f567c58cb0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.188+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f40317b0 con 0x55f567c58cb0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb8ec0213f0 con 0x55f567c58cb0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 10 client.16701 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 20 client.16701 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567c68ba0 0x55f567c6af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 4 client.16701 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f567bbfa20 con 0x55f567c68ba0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.189+0000 7fb903fff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567c68ba0 0x55f567c6af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.190+0000 7fb903fff640 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567c68ba0 0x55f567c6af90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7b0b40165348cba server_cookie=7de94a34c66a5bb8 in_seq=0 out_seq=0 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.190+0000 7fb8e37fe640 10 client.16701 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:22.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.190+0000 7fb8e37fe640 1 -- 172.21.15.78:0/651706349 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f567bbfa20 con 0x55f567c68ba0 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.190+0000 7fb8e37fe640 10 client.16701 handle_command_reply: tid=0 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.190+0000 7fb90d412740 1 client.16701 shutdown 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 msgr2=0x55f567c56610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec013de0 tx=0x7fb8ec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 shutdown_connections 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567c68ba0 0x55f567c6af90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c56610 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 --2- 172.21.15.78:0/651706349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4daf0 0x55f567c55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb8e37fe640 0 client.16701 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 >> 172.21.15.78:0/651706349 conn(0x55f567c4a060 msgr2=0x55f567c4fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 shutdown_connections 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 1 -- 172.21.15.78:0/651706349 wait complete. 2024-09-16T01:22:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.191+0000 7fb90d412740 20 client.16701 trim_cache size 0 max 16384 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 Processor -- start 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 -- start start 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c70800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c70d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b3af0 con 0x55f567c4daf0 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b13c0 con 0x55f567c58cb0 2024-09-16T01:22:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb90d412740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041af230 con 0x55f567c4e410 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58534/0 (socket says 172.21.15.78:58534) 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 -- 172.21.15.78:0/382269273 learned_addr learned my addr 172.21.15.78:0/382269273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb903fff640 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c70800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 -- 172.21.15.78:0/382269273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 msgr2=0x55f567c70800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c70800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.205+0000 7fb8fbfff640 1 -- 172.21.15.78:0/382269273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 msgr2=0x55f567c70d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c70d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8fbfff640 1 -- 172.21.15.78:0/382269273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f567c58720 con 0x55f567c4daf0 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec013b10 tx=0x7fb8ec00b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cde2abfe8232c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/382269273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec026070 con 0x55f567c4daf0 2024-09-16T01:22:22.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/382269273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ec013bb0 con 0x55f567c4daf0 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb8f9ffb640 1 -- 172.21.15.78:0/382269273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec004030 con 0x55f567c4daf0 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 -- 172.21.15.78:0/382269273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 msgr2=0x55f567c56440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec013b10 tx=0x7fb8ec00b4e0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 -- 172.21.15.78:0/382269273 shutdown_connections 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c58cb0 0x55f567c70d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4e410 0x55f567c70800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 --2- 172.21.15.78:0/382269273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c4daf0 0x55f567c56440 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.206+0000 7fb90d412740 1 -- 172.21.15.78:0/382269273 >> 172.21.15.78:0/382269273 conn(0x55f567c4a060 msgr2=0x55f567c57c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- 172.21.15.78:0/382269273 shutdown_connections 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- 172.21.15.78:0/382269273 wait complete. 2024-09-16T01:22:22.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- start start 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 Processor -- start 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4e410 0x55f567c5c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041af880 con 0x55f567c58cb0 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041b0cf0 con 0x55f567c4e410 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041aff50 con 0x55f567c4daf0 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb90d412740 10 client.0 _ll_register_callbacks cb 0x7fb8ec00bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58540/0 (socket says 172.21.15.78:58540) 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb8fbfff640 1 -- 172.21.15.78:0/1846690215 learned_addr learned my addr 172.21.15.78:0/1846690215 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:22.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.207+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb903fff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4e410 0x55f567c5c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 -- 172.21.15.78:0/1846690215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 msgr2=0x55f567c612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 -- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4e410 msgr2=0x55f567c5c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4e410 0x55f567c5c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 -- 172.21.15.78:0/1846690215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f567c452d0 con 0x55f567c58cb0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8fbfff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00bc40 tx=0x7fb8ec0215c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed09ebde98ae73f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec026070 con 0x55f567c58cb0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fb8ec021c10 con 0x55f567c58cb0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8ec013bb0 con 0x55f567c58cb0 2024-09-16T01:22:22.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8ec013260 con 0x55f567c58cb0 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb90d412740 10 client.16707 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f567c50940 con 0x55f567c58cb0 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ec013bb0 con 0x55f567c58cb0 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb8ec039020 con 0x55f567c58cb0 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.208+0000 7fb90d412740 10 client.16707 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb90d412740 10 client.16707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb90d412740 20 client.16707 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567cd3940 0x55f567cd5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb90d412740 4 client.16707 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f567bb9d70 con 0x55f567cd3940 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb903fff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567cd3940 0x55f567cd5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb8e37fe640 10 client.16707 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:22.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.209+0000 7fb903fff640 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567cd3940 0x55f567cd5d30 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a3fab244afb930a server_cookie=aa39b3e9bbc139cb in_seq=0 out_seq=0 2024-09-16T01:22:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.217+0000 7fb8e37fe640 1 -- 172.21.15.78:0/1846690215 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f567bb9d70 con 0x55f567cd3940 2024-09-16T01:22:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.217+0000 7fb8e37fe640 10 client.16707 handle_command_reply: tid=0 2024-09-16T01:22:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.217+0000 7fb90d412740 1 client.16707 shutdown 2024-09-16T01:22:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 msgr2=0x55f567c5cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00bc40 tx=0x7fb8ec0215c0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 shutdown_connections 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f567cd3940 0x55f567cd5d30 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f567c58cb0 0x55f567c5cb20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f567c4e410 0x55f567c5c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 --2- 172.21.15.78:0/1846690215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f567c4daf0 0x55f567c612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb8e37fe640 0 client.16707 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 >> 172.21.15.78:0/1846690215 conn(0x55f567c4a060 msgr2=0x55f567c57c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 shutdown_connections 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 1 -- 172.21.15.78:0/1846690215 wait complete. 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90d412740 20 client.16707 trim_cache size 0 max 16384 2024-09-16T01:22:22.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:22:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:22:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:22:22.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 msgr2=0x7fb8f00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 0x7fb8f00a59a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc09bfe0 tx=0x7fb8fc085040 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 msgr2=0x7fb904101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00cad0 tx=0x7fb8fc00cf90 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 shutdown_connections 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fb8f00a34e0 0x7fb8f00a59a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb90410c0d0 0x7fb904101350 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9041035a0 0x7fb904100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 --2- 172.21.15.78:0/2240275223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb904102c80 0x7fb9041008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 >> 172.21.15.78:0/2240275223 conn(0x7fb904077990 msgr2=0x7fb9040fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:22.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 shutdown_connections 2024-09-16T01:22:22.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:22.218+0000 7fb90a677640 1 -- 172.21.15.78:0/2240275223 wait complete. 2024-09-16T01:22:22.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:22 smithi078 ceph-mon[30937]: pgmap v362: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 214 KiB/s wr, 33 op/s 2024-09-16T01:22:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:22 smithi148 ceph-mon[35030]: pgmap v362: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 214 KiB/s wr, 33 op/s 2024-09-16T01:22:22.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:22:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:22 smithi193 ceph-mon[35701]: pgmap v362: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 214 KiB/s wr, 33 op/s 2024-09-16T01:22:23.144 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- 172.21.15.78:0/3300922946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa80810e440 msgr2=0x7fa808110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- 172.21.15.78:0/3300922946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa80810e440 0x7fa808110830 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa800009ca0 tx=0x7fa80002a950 comp rx=0 tx=0).stop 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- 172.21.15.78:0/3300922946 shutdown_connections 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- 172.21.15.78:0/3300922946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa80810e440 0x7fa808110830 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- 172.21.15.78:0/3300922946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074e30 0x7fa80810df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- 172.21.15.78:0/3300922946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808074510 0x7fa8080748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- 172.21.15.78:0/3300922946 >> 172.21.15.78:0/3300922946 conn(0x7fa8080fd8c0 msgr2=0x7fa8080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- 172.21.15.78:0/3300922946 shutdown_connections 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- 172.21.15.78:0/3300922946 wait complete. 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 Processor -- start 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 -- start start 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074510 0x7fa80819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.697+0000 7fa80f899640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa80810e440 0x7fa80819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80f899640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa808114160 con 0x7fa808074e30 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80f899640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa808113fe0 con 0x7fa808074510 2024-09-16T01:22:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80f899640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8081142e0 con 0x7fa80810e440 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58558/0 (socket says 172.21.15.78:58558) 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 -- 172.21.15.78:0/260894315 learned_addr learned my addr 172.21.15.78:0/260894315 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80de36640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa80810e440 0x7fa80819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80d635640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074510 0x7fa80819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 -- 172.21.15.78:0/260894315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa80810e440 msgr2=0x7fa80819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa80810e440 0x7fa80819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 -- 172.21.15.78:0/260894315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074510 msgr2=0x7fa80819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074510 0x7fa80819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 -- 172.21.15.78:0/260894315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80819f6d0 con 0x7fa808074e30 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80ce34640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800d950 tx=0x7fa7f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8017070 con 0x7fa808074e30 2024-09-16T01:22:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f8009c30 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800f6b0 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80819f9c0 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.698+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a72a0 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.699+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8080748f0 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.702+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fa7f80109d0 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.702+0000 7fa7fe7fc640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 0x7fa7f40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.702+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fa7f80cfb50 con 0x7fa808074e30 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.702+0000 7fa80d635640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 0x7fa7f40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.703+0000 7fa80d635640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 0x7fa7f40a58f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa804005fd0 tx=0x7fa80400b390 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:23.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.706+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa7f8009de0 con 0x7fa808074e30 2024-09-16T01:22:23.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.919+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa80819ba80 con 0x7fa808074e30 2024-09-16T01:22:23.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.920+0000 7fa7fe7fc640 1 -- 172.21.15.78:0/260894315 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fa7f809bb70 con 0x7fa808074e30 2024-09-16T01:22:23.922 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:22:23.922 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:22:23.922 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:22:23.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 msgr2=0x7fa7f40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 0x7fa7f40a58f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa804005fd0 tx=0x7fa80400b390 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 msgr2=0x7fa80819ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800d950 tx=0x7fa7f800de10 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 shutdown_connections 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa7f40a3430 0x7fa7f40a58f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa80810e440 0x7fa80819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa808074e30 0x7fa80819ac60 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 --2- 172.21.15.78:0/260894315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa808074510 0x7fa80819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 >> 172.21.15.78:0/260894315 conn(0x7fa8080fd8c0 msgr2=0x7fa8080fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 shutdown_connections 2024-09-16T01:22:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:23.923+0000 7fa80f899640 1 -- 172.21.15.78:0/260894315 wait complete. 2024-09-16T01:22:24.812 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 17ca30df --release --await' 2024-09-16T01:22:24.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 17ca30df --release --await 2024-09-16T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:24 smithi078 ceph-mon[30937]: pgmap v363: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-09-16T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:24 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/260894315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:24 smithi148 ceph-mon[35030]: pgmap v363: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-09-16T01:22:24.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:24 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/260894315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:24 smithi193 ceph-mon[35701]: pgmap v363: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-09-16T01:22:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:24 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/260894315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:22:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 -- 172.21.15.78:0/1223531809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914102790 msgr2=0x7f59141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 --2- 172.21.15.78:0/1223531809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914102790 0x7f59141067e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5904009900 tx=0x7f590402a410 comp rx=0 tx=0).stop 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 -- 172.21.15.78:0/1223531809 shutdown_connections 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 --2- 172.21.15.78:0/1223531809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914106f10 0x7f591410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 --2- 172.21.15.78:0/1223531809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914102790 0x7f59141067e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 --2- 172.21.15.78:0/1223531809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914101de0 0x7f59141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 -- 172.21.15.78:0/1223531809 >> 172.21.15.78:0/1223531809 conn(0x7f59140fd8c0 msgr2=0x7f59140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 -- 172.21.15.78:0/1223531809 shutdown_connections 2024-09-16T01:22:25.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.657+0000 7f591b6f0640 1 -- 172.21.15.78:0/1223531809 wait complete. 2024-09-16T01:22:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 Processor -- start 2024-09-16T01:22:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 -- start start 2024-09-16T01:22:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 0x7f591419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914106f10 0x7f591419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5914111f90 con 0x7f5914101de0 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5914111e10 con 0x7f5914102790 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591b6f0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5914112110 con 0x7f5914106f10 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45918/0 (socket says 172.21.15.78:45918) 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 -- 172.21.15.78:0/2050565481 learned_addr learned my addr 172.21.15.78:0/2050565481 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f5919c8d640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914106f10 0x7f591419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 0x7f591419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 -- 172.21.15.78:0/2050565481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914106f10 msgr2=0x7f591419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914106f10 0x7f591419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 -- 172.21.15.78:0/2050565481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 msgr2=0x7f591419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 0x7f591419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f591948c640 1 -- 172.21.15.78:0/2050565481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591419f6e0 con 0x7f5914101de0 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.658+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 0x7f591419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f591948c640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f591000b4a0 tx=0x7f591000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f590a7fc640 1 -- 172.21.15.78:0/2050565481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910014070 con 0x7f5914101de0 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f590a7fc640 1 -- 172.21.15.78:0/2050565481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5910010af0 con 0x7f5914101de0 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591419f9d0 con 0x7f5914101de0 2024-09-16T01:22:25.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f590a7fc640 1 -- 172.21.15.78:0/2050565481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910017d10 con 0x7f5914101de0 2024-09-16T01:22:25.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.659+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59141a72b0 con 0x7f5914101de0 2024-09-16T01:22:25.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.661+0000 7f590a7fc640 1 -- 172.21.15.78:0/2050565481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f5910017070 con 0x7f5914101de0 2024-09-16T01:22:25.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.661+0000 7f590a7fc640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 0x7f59000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.662+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 0x7f59000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.662+0000 7f590a7fc640 1 -- 172.21.15.78:0/2050565481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f59100cf4f0 con 0x7f5914101de0 2024-09-16T01:22:25.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.662+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 0x7f59000a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5904000c00 tx=0x7f5904032d50 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 Processor -- start 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 -- start start 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563574b97620 con 0x563574c32060 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563574b3ee50 con 0x563574c32980 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591dac9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563574b935e0 con 0x563574c3d240 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45922/0 (socket says 172.21.15.78:45922) 2024-09-16T01:22:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 -- 172.21.15.78:0/3769173569 learned_addr learned my addr 172.21.15.78:0/3769173569 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f5919c8d640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f5918c8b640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 -- 172.21.15.78:0/3769173569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 msgr2=0x563574c3f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 -- 172.21.15.78:0/3769173569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 msgr2=0x563574c3cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 -- 172.21.15.78:0/3769173569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563574b2fd40 con 0x563574c32060 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f5919c8d640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f5918c8b640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.673+0000 7f591948c640 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f591009b480 tx=0x7f591009bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13612a2bd2ab0a40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f58ff7fe640 1 -- 172.21.15.78:0/3769173569 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591001e070 con 0x563574c32060 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f58ff7fe640 1 -- 172.21.15.78:0/3769173569 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59100a4160 con 0x563574c32060 2024-09-16T01:22:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f58ff7fe640 1 -- 172.21.15.78:0/3769173569 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910010ca0 con 0x563574c32060 2024-09-16T01:22:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 -- 172.21.15.78:0/3769173569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 msgr2=0x563574c32440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f591009b480 tx=0x7f591009bbf0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 -- 172.21.15.78:0/3769173569 shutdown_connections 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 --2- 172.21.15.78:0/3769173569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c32440 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 -- 172.21.15.78:0/3769173569 >> 172.21.15.78:0/3769173569 conn(0x563574c2e5f0 msgr2=0x563574c2e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 -- 172.21.15.78:0/3769173569 shutdown_connections 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.674+0000 7f591dac9740 1 -- 172.21.15.78:0/3769173569 wait complete. 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 -- start start 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 Processor -- start 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x563574c3fe10 con 0x563574c32060 2024-09-16T01:22:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563574b1ff10 con 0x563574c32980 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563574c3ff90 con 0x563574c3d240 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591dac9740 10 client.0 _ll_register_callbacks cb 0x563574b43db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45928/0 (socket says 172.21.15.78:45928) 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 -- 172.21.15.78:0/4227119117 learned_addr learned my addr 172.21.15.78:0/4227119117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f5919c8d640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 -- 172.21.15.78:0/4227119117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 msgr2=0x563574c3abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 -- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 msgr2=0x563574c3a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f591948c640 1 -- 172.21.15.78:0/4227119117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635748a6580 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.675+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591948c640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f591009b1f0 tx=0x7f5910017590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed9de50e7c024e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591001e050 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f59100cf170 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59100a4160 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5910097cd0 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 10 client.16725 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563574970960 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59100a4160 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f591009a940 con 0x563574c32060 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 10 client.16725 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 10 client.16725 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 20 client.16725 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c4cc00 0x563574c4eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 4 client.16725 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.676+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563574ba4b90 con 0x563574c4cc00 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.677+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c4cc00 0x563574c4eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.677+0000 7f58fcff9640 10 client.16725 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.677+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c4cc00 0x563574c4eff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8104d9eec8baeb server_cookie=ff33219a6efd9e11 in_seq=0 out_seq=0 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f58fcff9640 1 -- 172.21.15.78:0/4227119117 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563574ba4b90 con 0x563574c4cc00 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f58fcff9640 10 client.16725 handle_command_reply: tid=0 2024-09-16T01:22:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 client.16725 shutdown 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 msgr2=0x563574c3a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f591009b1f0 tx=0x7f5910017590 comp rx=0 tx=0).stop 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 shutdown_connections 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c4cc00 0x563574c4eff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c3d240 0x563574c3abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c3a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f58fcff9640 0 client.16725 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 --2- 172.21.15.78:0/4227119117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c3a170 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 >> 172.21.15.78:0/4227119117 conn(0x563574c2e5f0 msgr2=0x563574c34320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 shutdown_connections 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.678+0000 7f591dac9740 1 -- 172.21.15.78:0/4227119117 wait complete. 2024-09-16T01:22:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.679+0000 7f591dac9740 20 client.16725 trim_cache size 0 max 16384 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 Processor -- start 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 -- start start 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 0x563574c52010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c373d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b3cb0 con 0x563574c32060 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b1580 con 0x563574c32980 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f591dac9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59141af3f0 con 0x563574c2e5f0 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f5918c8b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f5918c8b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45938/0 (socket says 172.21.15.78:45938) 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.695+0000 7f5918c8b640 1 -- 172.21.15.78:0/2246076377 learned_addr learned my addr 172.21.15.78:0/2246076377 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f591948c640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 0x563574c52010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5919c8d640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 -- 172.21.15.78:0/2246076377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 msgr2=0x563574c52010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 0x563574c52010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 -- 172.21.15.78:0/2246076377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 msgr2=0x563574c373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 -- 172.21.15.78:0/2246076377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563574c2f280 con 0x563574c32060 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f591948c640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 0x563574c52010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5919c8d640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f5918c8b640 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5904009230 tx=0x7f590403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c24d4431b4454f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f58feffd640 1 -- 172.21.15.78:0/2246076377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904057070 con 0x563574c32060 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f58feffd640 1 -- 172.21.15.78:0/2246076377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904041050 con 0x563574c32060 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.696+0000 7f58feffd640 1 -- 172.21.15.78:0/2246076377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904035050 con 0x563574c32060 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- 172.21.15.78:0/2246076377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 msgr2=0x563574c52550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5904009230 tx=0x7f590403b000 comp rx=0 tx=0).stop 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- 172.21.15.78:0/2246076377 shutdown_connections 2024-09-16T01:22:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c32980 0x563574c373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52550 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- 172.21.15.78:0/2246076377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c2e5f0 0x563574c52010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- 172.21.15.78:0/2246076377 >> 172.21.15.78:0/2246076377 conn(0x563574c35b60 msgr2=0x563574c38a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- 172.21.15.78:0/2246076377 shutdown_connections 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- 172.21.15.78:0/2246076377 wait complete. 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- start start 2024-09-16T01:22:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 Processor -- start 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c2e5f0 0x563574c52200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c32980 0x563574c505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f59141afa40 con 0x563574c32060 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b0eb0 con 0x563574c2e5f0 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b0110 con 0x563574c32980 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.697+0000 7f591dac9740 10 client.0 _ll_register_callbacks cb 0x7f59100997f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45944/0 (socket says 172.21.15.78:45944) 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 -- 172.21.15.78:0/4212685817 learned_addr learned my addr 172.21.15.78:0/4212685817 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f591948c640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c2e5f0 0x563574c52200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 -- 172.21.15.78:0/4212685817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c32980 msgr2=0x563574c505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c32980 0x563574c505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 -- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c2e5f0 msgr2=0x563574c52200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c2e5f0 0x563574c52200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 -- 172.21.15.78:0/4212685817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563574c50e20 con 0x563574c32060 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f5918c8b640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f59040318b0 tx=0x7f5904031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca28bc2548491d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f58fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904057070 con 0x563574c32060 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f5904005250 con 0x563574c32060 2024-09-16T01:22:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5904041050 con 0x563574c32060 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5904065050 con 0x563574c32060 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f591dac9740 10 client.16731 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563574cb8060 con 0x563574c32060 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.698+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904041050 con 0x563574c32060 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f59040037b0 con 0x563574c32060 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 10 client.16731 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 10 client.16731 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 20 client.16731 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c3d240 0x563574cba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 4 client.16731 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "17ca30df", "release": true, "await": true}] 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "17ca30df", "release": true, "await": true}) -- 0x563574b9f0a0 con 0x563574c3d240 2024-09-16T01:22:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591948c640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c3d240 0x563574cba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f58fcff9640 10 client.16731 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.699+0000 7f591948c640 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c3d240 0x563574cba7f0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e7d1459d1945e21 server_cookie=77238d8da1b815da in_seq=0 out_seq=0 2024-09-16T01:22:25.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.760+0000 7f58fcff9640 1 -- 172.21.15.78:0/4212685817 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563574b9f0a0 con 0x563574c3d240 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.760+0000 7f58fcff9640 10 client.16731 handle_command_reply: tid=0 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 client.16731 shutdown 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 msgr2=0x563574c52740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f59040318b0 tx=0x7f5904031a60 comp rx=0 tx=0).stop 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 shutdown_connections 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x563574c3d240 0x563574cba7f0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563574c32980 0x563574c505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563574c32060 0x563574c52740 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f58fcff9640 0 client.16731 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 --2- 172.21.15.78:0/4212685817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563574c2e5f0 0x563574c52200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 >> 172.21.15.78:0/4212685817 conn(0x563574c35b60 msgr2=0x563574c48a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 shutdown_connections 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 1 -- 172.21.15.78:0/4212685817 wait complete. 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:22:25.767 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 44, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "17ca30df": { 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 44, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 141.8, 2024-09-16T01:22:25.768 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:22:25.769 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:22:25.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.761+0000 7f591dac9740 20 client.16731 trim_cache size 0 max 16384 2024-09-16T01:22:25.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 msgr2=0x7f59000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 0x7f59000a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5904000c00 tx=0x7f5904032d50 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 msgr2=0x7f591419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f591000b4a0 tx=0x7f591000b960 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 shutdown_connections 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59000a33b0 0x7f59000a5870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5914106f10 0x7f591419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5914102790 0x7f591419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 --2- 172.21.15.78:0/2050565481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5914101de0 0x7f591419a730 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 >> 172.21.15.78:0/2050565481 conn(0x7f59140fd8c0 msgr2=0x7f59140fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 shutdown_connections 2024-09-16T01:22:25.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:22:25.762+0000 7f591b6f0640 1 -- 172.21.15.78:0/2050565481 wait complete. 2024-09-16T01:22:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 670.0 s, 24.6 tps, lat 11216.946 ms stddev 9899.419 2024-09-16T01:22:26.545 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '17ca30df', total seconds elapsed: 25.5 2024-09-16T01:22:26.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:26 smithi078 ceph-mon[30937]: pgmap v364: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 43 KiB/s wr, 12 op/s 2024-09-16T01:22:26.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:26 smithi148 ceph-mon[35030]: pgmap v364: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 43 KiB/s wr, 12 op/s 2024-09-16T01:22:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:26 smithi193 ceph-mon[35701]: pgmap v364: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 43 KiB/s wr, 12 op/s 2024-09-16T01:22:28.848 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:28 smithi193 ceph-mon[35701]: pgmap v365: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 128 KiB/s wr, 23 op/s 2024-09-16T01:22:28.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:28 smithi148 ceph-mon[35030]: pgmap v365: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 128 KiB/s wr, 23 op/s 2024-09-16T01:22:29.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:28 smithi078 ceph-mon[30937]: pgmap v365: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 128 KiB/s wr, 23 op/s 2024-09-16T01:22:30.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:30 smithi148 ceph-mon[35030]: pgmap v366: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-16T01:22:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 675.0 s, 625.2 tps, lat 350.445 ms stddev 1996.361 2024-09-16T01:22:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:30 smithi078 ceph-mon[30937]: pgmap v366: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-16T01:22:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:30 smithi193 ceph-mon[35701]: pgmap v366: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-09-16T01:22:32.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:32 smithi148 ceph-mon[35030]: pgmap v367: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-09-16T01:22:33.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:32 smithi078 ceph-mon[30937]: pgmap v367: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-09-16T01:22:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:32 smithi193 ceph-mon[35701]: pgmap v367: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.6 MiB/s wr, 201 op/s 2024-09-16T01:22:34.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:34 smithi148 ceph-mon[35030]: pgmap v368: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.4 MiB/s wr, 257 op/s 2024-09-16T01:22:35.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:34 smithi078 ceph-mon[30937]: pgmap v368: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.4 MiB/s wr, 257 op/s 2024-09-16T01:22:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:34 smithi193 ceph-mon[35701]: pgmap v368: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.4 MiB/s wr, 257 op/s 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 -- 172.21.15.78:0/3109517418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac101d70 msgr2=0x7f02ac1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 --2- 172.21.15.78:0/3109517418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac101d70 0x7f02ac1021b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009930 tx=0x7f02a002a420 comp rx=0 tx=0).stop 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 -- 172.21.15.78:0/3109517418 shutdown_connections 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 --2- 172.21.15.78:0/3109517418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac1026f0 0x7f02ac10d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 --2- 172.21.15.78:0/3109517418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac101d70 0x7f02ac1021b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 --2- 172.21.15.78:0/3109517418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.351+0000 7f02b30d0640 1 -- 172.21.15.78:0/3109517418 >> 172.21.15.78:0/3109517418 conn(0x7f02ac0770f0 msgr2=0x7f02ac0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.352+0000 7f02b30d0640 1 -- 172.21.15.78:0/3109517418 shutdown_connections 2024-09-16T01:22:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.352+0000 7f02b30d0640 1 -- 172.21.15.78:0/3109517418 wait complete. 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.352+0000 7f02b30d0640 1 Processor -- start 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.352+0000 7f02b30d0640 1 -- start start 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac101d70 0x7f02ac1a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac1026f0 0x7f02ac10bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f5b0 con 0x7f02ac100490 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f430 con 0x7f02ac101d70 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b30d0640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f730 con 0x7f02ac1026f0 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55488/0 (socket says 172.21.15.78:55488) 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 -- 172.21.15.78:0/734036735 learned_addr learned my addr 172.21.15.78:0/734036735 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b166d640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac1026f0 0x7f02ac10bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02abfff640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac101d70 0x7f02ac1a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 -- 172.21.15.78:0/734036735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac1026f0 msgr2=0x7f02ac10bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac1026f0 0x7f02ac10bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 -- 172.21.15.78:0/734036735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac101d70 msgr2=0x7f02ac1a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac101d70 0x7f02ac1a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 -- 172.21.15.78:0/734036735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac10c4b0 con 0x7f02ac100490 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.353+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f029800d950 tx=0x7f029800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.354+0000 7f02a9ffb640 1 -- 172.21.15.78:0/734036735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298017070 con 0x7f02ac100490 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.354+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac10c7a0 con 0x7f02ac100490 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.354+0000 7f02a9ffb640 1 -- 172.21.15.78:0/734036735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f029800de80 con 0x7f02ac100490 2024-09-16T01:22:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.354+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac1ab800 con 0x7f02ac100490 2024-09-16T01:22:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.355+0000 7f02a9ffb640 1 -- 172.21.15.78:0/734036735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298010660 con 0x7f02ac100490 2024-09-16T01:22:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.359+0000 7f02b5e66740 1 Processor -- start 2024-09-16T01:22:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.359+0000 7f02a9ffb640 1 -- 172.21.15.78:0/734036735 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f0298010800 con 0x7f02ac100490 2024-09-16T01:22:35.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 -- start start 2024-09-16T01:22:35.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 0x55736967aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 0x557369685770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 0x5573696880a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5573695df2f0 con 0x55736967b410 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557369587270 con 0x55736967aaf0 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b5e66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5573695db2b0 con 0x557369685cb0 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b166d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 0x55736967aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b166d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 0x55736967aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45266/0 (socket says 172.21.15.78:45266) 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b166d640 1 -- 172.21.15.78:0/1495480379 learned_addr learned my addr 172.21.15.78:0/1495480379 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02abfff640 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 0x5573696880a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 0x557369685770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.360+0000 7f02a9ffb640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 0x7f029c0a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02a9ffb640 1 -- 172.21.15.78:0/734036735 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f0298010460 con 0x7f02ac100490 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 -- 172.21.15.78:0/1495480379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 msgr2=0x55736967aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02b166d640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 0x7f029c0a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 0x55736967aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 -- 172.21.15.78:0/1495480379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 msgr2=0x557369685770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 0x557369685770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 -- 172.21.15.78:0/1495480379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557369586340 con 0x557369685cb0 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 0x557369685770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:22:35.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02abfff640 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 0x5573696880a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009850 tx=0x7f02a0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d07e237ad0bd4af server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f02b166d640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 0x7f029c0a5b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009fd0 tx=0x7f02a4004ab0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f0296ffd640 1 -- 172.21.15.78:0/1495480379 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0035040 con 0x557369685cb0 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f0296ffd640 1 -- 172.21.15.78:0/1495480379 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a0004430 con 0x557369685cb0 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.361+0000 7f0296ffd640 1 -- 172.21.15.78:0/1495480379 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a00035f0 con 0x557369685cb0 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.362+0000 7f02b5e66740 1 -- 172.21.15.78:0/1495480379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 msgr2=0x5573696880a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.362+0000 7f02b5e66740 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 0x5573696880a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009850 tx=0x7f02a0004040 comp rx=0 tx=0).stop 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 -- 172.21.15.78:0/1495480379 shutdown_connections 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557369685cb0 0x5573696880a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967b410 0x557369685770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 --2- 172.21.15.78:0/1495480379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967aaf0 0x55736967aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 -- 172.21.15.78:0/1495480379 >> 172.21.15.78:0/1495480379 conn(0x557369677060 msgr2=0x557369677440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 -- 172.21.15.78:0/1495480379 shutdown_connections 2024-09-16T01:22:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 -- 172.21.15.78:0/1495480379 wait complete. 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 -- start start 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.363+0000 7f02b5e66740 1 Processor -- start 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x557369682c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369683170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x5573696836b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557369688880 con 0x557369685cb0 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b166d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x5573696836b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b166d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x5573696836b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55512/0 (socket says 172.21.15.78:55512) 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b166d640 1 -- 172.21.15.78:0/213534523 learned_addr learned my addr 172.21.15.78:0/213534523 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x557369682c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557369568290 con 0x55736967b410 2024-09-16T01:22:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557369688a00 con 0x55736967aaf0 2024-09-16T01:22:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.364+0000 7f02b5e66740 10 client.0 _ll_register_callbacks cb 0x557369599140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.365+0000 7f02abfff640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369683170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 -- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 msgr2=0x557369683170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369683170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 -- 172.21.15.78:0/213534523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 msgr2=0x5573696836b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x5573696836b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573692f2210 con 0x55736967aaf0 2024-09-16T01:22:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02abfff640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369683170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:22:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x557369682c30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02980cc530 tx=0x7f029809b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22f8d9f673788a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f028bfff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029809b150 con 0x55736967aaf0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.369+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f02980cc7b0 con 0x55736967aaf0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.370+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a0004430 con 0x55736967aaf0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.370+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f029809f500 con 0x55736967aaf0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.370+0000 7f02b5e66740 10 client.26110 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.370+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5573693be620 con 0x55736967aaf0 2024-09-16T01:22:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02980977c0 con 0x55736967aaf0 2024-09-16T01:22:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0298010af0 con 0x55736967aaf0 2024-09-16T01:22:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 10 client.26110 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 10 client.26110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 20 client.26110 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369695c40 0x557369698030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 4 client.26110 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f02a0004430 con 0x557369695c40 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.371+0000 7f02abfff640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369695c40 0x557369698030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.372+0000 7f028bfff640 10 client.26110 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.372+0000 7f02abfff640 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369695c40 0x557369698030 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4df8e55c6751f8d1 server_cookie=ac355eed9703b6fc in_seq=0 out_seq=0 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.372+0000 7f028bfff640 1 -- 172.21.15.78:0/213534523 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f02a0004430 con 0x557369695c40 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.372+0000 7f028bfff640 10 client.26110 handle_command_reply: tid=0 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 client.26110 shutdown 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 msgr2=0x557369682c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x557369682c30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02980cc530 tx=0x7f029809b850 comp rx=0 tx=0).stop 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 shutdown_connections 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369695c40 0x557369698030 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x5573696836b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369683170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 --2- 172.21.15.78:0/213534523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x557369682c30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f028bfff640 0 client.26110 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 >> 172.21.15.78:0/213534523 conn(0x557369677060 msgr2=0x55736967cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 shutdown_connections 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 1 -- 172.21.15.78:0/213534523 wait complete. 2024-09-16T01:22:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.373+0000 7f02b5e66740 20 client.26110 trim_cache size 0 max 16384 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.396+0000 7f02b5e66740 1 Processor -- start 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 -- start start 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x5573696834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369684ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x55736969d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b80c0 con 0x557369685cb0 2024-09-16T01:22:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b5990 con 0x55736967b410 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b5e66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b3800 con 0x55736967aaf0 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x5573696834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x5573696834e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:41784/0 (socket says 172.21.15.78:41784) 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b0e6c640 1 -- 172.21.15.78:0/3258904799 learned_addr learned my addr 172.21.15.78:0/3258904799 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x55736969d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02abfff640 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369684ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 -- 172.21.15.78:0/3258904799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 msgr2=0x5573696834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x5573696834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 -- 172.21.15.78:0/3258904799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 msgr2=0x557369684ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369684ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.397+0000 7f02b166d640 1 -- 172.21.15.78:0/3258904799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573696722d0 con 0x557369685cb0 2024-09-16T01:22:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02b166d640 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x55736969d8a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f02a40049d0 tx=0x7f02a4015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f35f7d44378a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02967fc640 1 -- 172.21.15.78:0/3258904799 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4004b20 con 0x557369685cb0 2024-09-16T01:22:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02967fc640 1 -- 172.21.15.78:0/3258904799 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a4004cc0 con 0x557369685cb0 2024-09-16T01:22:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02967fc640 1 -- 172.21.15.78:0/3258904799 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a40057f0 con 0x557369685cb0 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02b5e66740 1 -- 172.21.15.78:0/3258904799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 msgr2=0x55736969d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.398+0000 7f02b5e66740 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x55736969d8a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f02a40049d0 tx=0x7f02a4015070 comp rx=0 tx=0).stop 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 -- 172.21.15.78:0/3258904799 shutdown_connections 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557369685cb0 0x55736969d8a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55736967b410 0x557369684ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 --2- 172.21.15.78:0/3258904799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967aaf0 0x5573696834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 -- 172.21.15.78:0/3258904799 >> 172.21.15.78:0/3258904799 conn(0x557369677060 msgr2=0x557369684490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 -- 172.21.15.78:0/3258904799 shutdown_connections 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 -- 172.21.15.78:0/3258904799 wait complete. 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 -- start start 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 Processor -- start 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.399+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 0x557369689210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 0x5573696fe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b3e50 con 0x55736967aaf0 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b52c0 con 0x557369685cb0 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac1b4520 con 0x55736967b410 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b5e66740 10 client.0 _ll_register_callbacks cb 0x7f029809aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 0x557369689210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02abfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 0x5573696fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b0e6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 0x557369689210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55524/0 (socket says 172.21.15.78:55524) 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02abfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:41792/0 (socket says 172.21.15.78:41792) 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b0e6c640 1 -- 172.21.15.78:0/2352099707 learned_addr learned my addr 172.21.15.78:0/2352099707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 -- 172.21.15.78:0/2352099707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 msgr2=0x557369689750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 -- 172.21.15.78:0/2352099707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 msgr2=0x557369689210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 0x557369689210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 -- 172.21.15.78:0/2352099707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573696fee20 con 0x557369685cb0 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02abfff640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f02b166d640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 0x5573696fe860 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f02a402c040 tx=0x7f02a4010c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b934b32f8108c55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.400+0000 7f028bfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:22:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4038020 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a4004cc0 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7f02a400fe00 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f02a4043050 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 10 client.26048 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557369700130 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a4004cc0 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f02a402a480 con 0x557369685cb0 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 10 client.26048 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 10 client.26048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 20 client.26048 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369700800 0x557369702bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 4 client.26048 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.401+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5573695eca20 con 0x557369700800 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.402+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369700800 0x557369702bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.402+0000 7f028bfff640 10 client.26048 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.402+0000 7f02b0e6c640 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369700800 0x557369702bf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31351372e24c2e04 server_cookie=5c960c4b8bacde3 in_seq=0 out_seq=0 2024-09-16T01:22:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.409+0000 7f028bfff640 1 -- 172.21.15.78:0/2352099707 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5573695eca20 con 0x557369700800 2024-09-16T01:22:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.409+0000 7f028bfff640 10 client.26048 handle_command_reply: tid=0 2024-09-16T01:22:35.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.409+0000 7f02b5e66740 1 client.26048 shutdown 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 msgr2=0x5573696fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 0x5573696fe860 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f02a402c040 tx=0x7f02a4010c30 comp rx=0 tx=0).stop 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 shutdown_connections 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557369700800 0x557369702bf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f028bfff640 0 client.26048 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557369685cb0 0x5573696fe860 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55736967b410 0x557369689750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 --2- 172.21.15.78:0/2352099707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55736967aaf0 0x557369689210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 >> 172.21.15.78:0/2352099707 conn(0x557369677060 msgr2=0x55736967cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.410+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 shutdown_connections 2024-09-16T01:22:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b5e66740 1 -- 172.21.15.78:0/2352099707 wait complete. 2024-09-16T01:22:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b5e66740 20 client.26048 trim_cache size 0 max 16384 2024-09-16T01:22:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:22:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:22:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:22:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 msgr2=0x7f029c0a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 0x7f029c0a5b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009fd0 tx=0x7f02a4004ab0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 msgr2=0x7f02ac1a47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f029800d950 tx=0x7f029800dd50 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 shutdown_connections 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f029c0a36b0 0x7f029c0a5b70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02ac1026f0 0x7f02ac10bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02ac101d70 0x7f02ac1a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 --2- 172.21.15.78:0/734036735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100490 0x7f02ac1a47d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.411+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 >> 172.21.15.78:0/734036735 conn(0x7f02ac0770f0 msgr2=0x7f02ac110ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.412+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 shutdown_connections 2024-09-16T01:22:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:35.412+0000 7f02b30d0640 1 -- 172.21.15.78:0/734036735 wait complete. 2024-09-16T01:22:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 680.0 s, 585.0 tps, lat 174.207 ms stddev 206.840 2024-09-16T01:22:36.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:36 smithi148 ceph-mon[35030]: pgmap v369: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.4 MiB/s wr, 346 op/s 2024-09-16T01:22:37.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:36 smithi078 ceph-mon[30937]: pgmap v369: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.4 MiB/s wr, 346 op/s 2024-09-16T01:22:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:36 smithi193 ceph-mon[35701]: pgmap v369: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.4 MiB/s wr, 346 op/s 2024-09-16T01:22:38.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:38 smithi148 ceph-mon[35030]: pgmap v370: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 6.4 MiB/s wr, 473 op/s 2024-09-16T01:22:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:38 smithi078 ceph-mon[30937]: pgmap v370: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 6.4 MiB/s wr, 473 op/s 2024-09-16T01:22:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:38 smithi193 ceph-mon[35701]: pgmap v370: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 6.4 MiB/s wr, 473 op/s 2024-09-16T01:22:40.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:40 smithi148 ceph-mon[35030]: pgmap v371: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 7.6 MiB/s wr, 526 op/s 2024-09-16T01:22:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 685.0 s, 776.2 tps, lat 124.072 ms stddev 150.386 2024-09-16T01:22:41.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:40 smithi078 ceph-mon[30937]: pgmap v371: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 7.6 MiB/s wr, 526 op/s 2024-09-16T01:22:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:40 smithi193 ceph-mon[35701]: pgmap v371: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 7.6 MiB/s wr, 526 op/s 2024-09-16T01:22:42.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:42 smithi148 ceph-mon[35030]: pgmap v372: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.7 MiB/s wr, 556 op/s 2024-09-16T01:22:43.003 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:42 smithi078 ceph-mon[30937]: pgmap v372: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.7 MiB/s wr, 556 op/s 2024-09-16T01:22:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:42 smithi193 ceph-mon[35701]: pgmap v372: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.7 MiB/s wr, 556 op/s 2024-09-16T01:22:44.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:44 smithi148 ceph-mon[35030]: pgmap v373: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-09-16T01:22:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:44 smithi078 ceph-mon[30937]: pgmap v373: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-09-16T01:22:45.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:44 smithi193 ceph-mon[35701]: pgmap v373: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 533 op/s 2024-09-16T01:22:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 690.0 s, 574.4 tps, lat 180.299 ms stddev 232.930 2024-09-16T01:22:46.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:46 smithi148 ceph-mon[35030]: pgmap v374: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-09-16T01:22:47.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:46 smithi078 ceph-mon[30937]: pgmap v374: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-09-16T01:22:47.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:46 smithi193 ceph-mon[35701]: pgmap v374: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 7.5 MiB/s wr, 607 op/s 2024-09-16T01:22:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 -- 172.21.15.78:0/3950785373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01095d0 msgr2=0x7fc9a0109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/3950785373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01095d0 0x7fc9a0109a10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc99001a940 tx=0x7fc99003b450 comp rx=0 tx=0).stop 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 -- 172.21.15.78:0/3950785373 shutdown_connections 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/3950785373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a011b140 0x7fc9a011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/3950785373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01095d0 0x7fc9a0109a10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/3950785373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9a01091f0 0x7fc9a0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 -- 172.21.15.78:0/3950785373 >> 172.21.15.78:0/3950785373 conn(0x7fc9a006ce00 msgr2=0x7fc9a006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 -- 172.21.15.78:0/3950785373 shutdown_connections 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 -- 172.21.15.78:0/3950785373 wait complete. 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.523+0000 7fc9a84f1640 1 Processor -- start 2024-09-16T01:22:48.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 -- start start 2024-09-16T01:22:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 0x7fc9a011a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 0x7fc9a01168c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9a011b140 0x7fc9a0116e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.530 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a0113300 con 0x7fc9a011b140 2024-09-16T01:22:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a0113180 con 0x7fc9a01091f0 2024-09-16T01:22:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a84f1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a0113480 con 0x7fc9a01095d0 2024-09-16T01:22:48.531 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a5a8c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 0x7fc9a01168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a5a8c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 0x7fc9a01168c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:56930/0 (socket says 172.21.15.78:56930) 2024-09-16T01:22:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a5a8c640 1 -- 172.21.15.78:0/858676220 learned_addr learned my addr 172.21.15.78:0/858676220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:48.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.524+0000 7fc9a628d640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 0x7fc9a011a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.529+0000 7fc9a628d640 1 -- 172.21.15.78:0/858676220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 msgr2=0x7fc9a01168c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.529+0000 7fc9a628d640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 0x7fc9a01168c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.529+0000 7fc9a628d640 1 -- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9a011b140 msgr2=0x7fc9a0116e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.529+0000 7fc9a628d640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9a011b140 0x7fc9a0116e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.529+0000 7fc9a628d640 1 -- 172.21.15.78:0/858676220 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a0117530 con 0x7fc9a01091f0 2024-09-16T01:22:48.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.535+0000 7fc9a628d640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 0x7fc9a011a7e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e9f0 tx=0x7fc99c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.535+0000 7fc9977fe640 1 -- 172.21.15.78:0/858676220 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00ccc0 con 0x7fc9a01091f0 2024-09-16T01:22:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.535+0000 7fc9977fe640 1 -- 172.21.15.78:0/858676220 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc99c010910 con 0x7fc9a01091f0 2024-09-16T01:22:48.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.535+0000 7fc9977fe640 1 -- 172.21.15.78:0/858676220 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c018960 con 0x7fc9a01091f0 2024-09-16T01:22:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.540+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a01ab070 con 0x7fc9a01091f0 2024-09-16T01:22:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.540+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a01ab420 con 0x7fc9a01091f0 2024-09-16T01:22:48.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.547+0000 7fc9ab28e740 1 Processor -- start 2024-09-16T01:22:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 -- start start 2024-09-16T01:22:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491caf0 0x55803491ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803492a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5580348812f0 con 0x55803491d410 2024-09-16T01:22:48.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558034829270 con 0x55803491caf0 2024-09-16T01:22:48.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.552+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55803487d2b0 con 0x558034927cb0 2024-09-16T01:22:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.554+0000 7fc9a628d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.554+0000 7fc9a628d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44386/0 (socket says 172.21.15.78:44386) 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.554+0000 7fc9a628d640 1 -- 172.21.15.78:0/1988912484 learned_addr learned my addr 172.21.15.78:0/1988912484 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.554+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491caf0 0x55803491ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803492a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 -- 172.21.15.78:0/1988912484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 msgr2=0x55803492a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803492a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 -- 172.21.15.78:0/1988912484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491caf0 msgr2=0x55803491ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491caf0 0x55803491ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 -- 172.21.15.78:0/1988912484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558034828340 con 0x55803491d410 2024-09-16T01:22:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.555+0000 7fc9a628d640 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0cd640 tx=0x7fc99c0d8660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada98a0b74a88926 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.556+0000 7fc994bfe640 1 -- 172.21.15.78:0/1988912484 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0e3070 con 0x55803491d410 2024-09-16T01:22:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.556+0000 7fc994bfe640 1 -- 172.21.15.78:0/1988912484 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc99c0d2550 con 0x55803491d410 2024-09-16T01:22:48.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.556+0000 7fc994bfe640 1 -- 172.21.15.78:0/1988912484 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0de9b0 con 0x55803491d410 2024-09-16T01:22:48.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.557+0000 7fc9ab28e740 1 -- 172.21.15.78:0/1988912484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 msgr2=0x558034927770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.557+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0cd640 tx=0x7fc99c0d8660 comp rx=0 tx=0).stop 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.564+0000 7fc9977fe640 1 -- 172.21.15.78:0/858676220 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fc99c01f020 con 0x7fc9a01091f0 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9ab28e740 1 -- 172.21.15.78:0/1988912484 shutdown_connections 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803492a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491d410 0x558034927770 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/1988912484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491caf0 0x55803491ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9ab28e740 1 -- 172.21.15.78:0/1988912484 >> 172.21.15.78:0/1988912484 conn(0x558034919060 msgr2=0x558034919440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9977fe640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 0x7fc98c0a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.565+0000 7fc9977fe640 1 -- 172.21.15.78:0/858676220 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fc99c0d0400 con 0x7fc9a01091f0 2024-09-16T01:22:48.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.567+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 0x7fc98c0a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.569+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 0x7fc98c0a5b00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55803491a3e0 tx=0x7fc990004820 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.571+0000 7fc9ab28e740 1 -- 172.21.15.78:0/1988912484 shutdown_connections 2024-09-16T01:22:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 -- 172.21.15.78:0/1988912484 wait complete. 2024-09-16T01:22:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 -- start start 2024-09-16T01:22:48.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 Processor -- start 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034924b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491d410 0x5580349250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55803492a880 con 0x55803491caf0 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55803480a290 con 0x558034927cb0 2024-09-16T01:22:48.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.572+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55803492aa00 con 0x55803491d410 2024-09-16T01:22:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.573+0000 7fc9ab28e740 10 client.0 _ll_register_callbacks cb 0x55803483b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.573+0000 7fc9a6a8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.573+0000 7fc9a6a8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36530/0 (socket says 172.21.15.78:36530) 2024-09-16T01:22:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.573+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4201474530 learned_addr learned my addr 172.21.15.78:0/4201474530 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:48.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.573+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491d410 0x5580349250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.577+0000 7fc9a628d640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034924b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.578+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4201474530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491d410 msgr2=0x5580349250c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:48.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.578+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491d410 0x5580349250c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.578+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4201474530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 msgr2=0x558034924b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:22:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.578+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034924b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.578+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4201474530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558034594210 con 0x558034927cb0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.580+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc998010960 tx=0x7fc998010e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ca47fd3d4f8cb17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.582+0000 7fc989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.582+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998017070 con 0x558034927cb0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.582+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc99c0d2550 con 0x558034927cb0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.582+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fc99800ed80 con 0x558034927cb0 2024-09-16T01:22:48.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.582+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc998003970 con 0x558034927cb0 2024-09-16T01:22:48.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.585+0000 7fc9ab28e740 10 client.26060 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.585+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558034660620 con 0x558034927cb0 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.586+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc99c0d2550 con 0x558034927cb0 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc998011440 con 0x558034927cb0 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 10 client.26060 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 20 client.26060 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558034937c80 0x55803493a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 4 client.26060 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:22:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.587+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55803488ea20 con 0x558034937c80 2024-09-16T01:22:48.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.588+0000 7fc9a628d640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558034937c80 0x55803493a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.600+0000 7fc9a628d640 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558034937c80 0x55803493a0f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2068721550925d9c server_cookie=5e9b9494f6031d1 in_seq=0 out_seq=0 2024-09-16T01:22:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.601+0000 7fc989ffb640 10 client.26060 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:48.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.604+0000 7fc989ffb640 1 -- 172.21.15.78:0/4201474530 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55803488ea20 con 0x558034937c80 2024-09-16T01:22:48.608 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.604+0000 7fc989ffb640 10 client.26060 handle_command_reply: tid=0 2024-09-16T01:22:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.606+0000 7fc9ab28e740 1 client.26060 shutdown 2024-09-16T01:22:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.610+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 msgr2=0x558034925600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.610+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc998010960 tx=0x7fc998010e20 comp rx=0 tx=0).stop 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 shutdown_connections 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x558034937c80 0x55803493a0f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558034927cb0 0x558034925600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491d410 0x5580349250c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4201474530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034924b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 >> 172.21.15.78:0/4201474530 conn(0x558034919060 msgr2=0x55803491edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.614+0000 7fc989ffb640 0 client.26060 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.617+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 shutdown_connections 2024-09-16T01:22:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.620+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4201474530 wait complete. 2024-09-16T01:22:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.620+0000 7fc9ab28e740 20 client.26060 trim_cache size 0 max 16384 2024-09-16T01:22:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.653+0000 7fc9ab28e740 1 Processor -- start 2024-09-16T01:22:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.653+0000 7fc9ab28e740 1 -- start start 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034925430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x55803493fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b8180 con 0x55803491caf0 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b5a50 con 0x55803491d410 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9a6a8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9a6a8e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:56952/0 (socket says 172.21.15.78:56952) 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4058815436 learned_addr learned my addr 172.21.15.78:0/4058815436 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.654+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b38c0 con 0x558034927cb0 2024-09-16T01:22:48.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.655+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4058815436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 msgr2=0x55803493fa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:22:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.655+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x55803493fa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.655+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4058815436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 msgr2=0x558034925430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:22:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.655+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034925430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.655+0000 7fc9a6a8e640 1 -- 172.21.15.78:0/4058815436 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558034918e90 con 0x558034927cb0 2024-09-16T01:22:48.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.656+0000 7fc9a6a8e640 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc998011860 tx=0x7fc998005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2539e76c6429d2b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.661+0000 7fc98bfff640 1 -- 172.21.15.78:0/4058815436 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99802d030 con 0x558034927cb0 2024-09-16T01:22:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.661+0000 7fc98bfff640 1 -- 172.21.15.78:0/4058815436 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc998011ad0 con 0x558034927cb0 2024-09-16T01:22:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.661+0000 7fc98bfff640 1 -- 172.21.15.78:0/4058815436 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998017040 con 0x558034927cb0 2024-09-16T01:22:48.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.662+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 msgr2=0x55803493ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.662+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc998011860 tx=0x7fc998005d30 comp rx=0 tx=0).stop 2024-09-16T01:22:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.666+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 shutdown_connections 2024-09-16T01:22:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.666+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558034927cb0 0x55803493ff40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.666+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x55803493fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.666+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/4058815436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55803491caf0 0x558034925430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.666+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 >> 172.21.15.78:0/4058815436 conn(0x558034919060 msgr2=0x558034926c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.669+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 shutdown_connections 2024-09-16T01:22:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.672+0000 7fc9ab28e740 1 -- 172.21.15.78:0/4058815436 wait complete. 2024-09-16T01:22:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.673+0000 7fc9ab28e740 1 -- start start 2024-09-16T01:22:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.673+0000 7fc9ab28e740 1 Processor -- start 2024-09-16T01:22:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x558034930200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558034927cb0 0x55803492cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b3f10 con 0x558034927cb0 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b5380 con 0x55803491d410 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01b45e0 con 0x55803491caf0 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9ab28e740 10 client.0 _ll_register_callbacks cb 0x7fc998009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9a628d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9a628d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:56956/0 (socket says 172.21.15.78:56956) 2024-09-16T01:22:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.678+0000 7fc9a628d640 1 -- 172.21.15.78:0/3683626672 learned_addr learned my addr 172.21.15.78:0/3683626672 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.679+0000 7fc9a628d640 1 -- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 msgr2=0x558034930200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.679+0000 7fc9a628d640 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x558034930200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.679+0000 7fc9a628d640 1 -- 172.21.15.78:0/3683626672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558034927cb0 msgr2=0x55803492cc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.679+0000 7fc9a628d640 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558034927cb0 0x55803492cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.679+0000 7fc9a628d640 1 -- 172.21.15.78:0/3683626672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55803491f8f0 con 0x55803491caf0 2024-09-16T01:22:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.680+0000 7fc9a628d640 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc99c000c00 tx=0x7fc99c0d0f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5c72699547bea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:22:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.680+0000 7fc989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:22:48.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.685+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c098530 con 0x55803491caf0 2024-09-16T01:22:48.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.685+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fc99c0d0170 con 0x55803491caf0 2024-09-16T01:22:48.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.687+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc998011ad0 con 0x55803491caf0 2024-09-16T01:22:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.692+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc99c010ac0 con 0x55803491caf0 2024-09-16T01:22:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.692+0000 7fc9ab28e740 10 client.26131 fetch_fsmap learned FSMap version 31 2024-09-16T01:22:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.692+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5580349a2e20 con 0x55803491caf0 2024-09-16T01:22:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc998011ad0 con 0x55803491caf0 2024-09-16T01:22:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc99c098900 con 0x55803491caf0 2024-09-16T01:22:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 10 client.26131 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:22:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 10 client.26131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:22:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 20 client.26131 populate_metadata read hostname 'smithi078' 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5580349a3110 0x5580349a5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 4 client.26131 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558034888d70 con 0x5580349a3110 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.693+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5580349a3110 0x5580349a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.694+0000 7fc9a5a8c640 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5580349a3110 0x5580349a5500 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29f7245b990c034b server_cookie=139db866e0f4588c in_seq=0 out_seq=0 2024-09-16T01:22:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.694+0000 7fc989ffb640 10 client.26131 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc989ffb640 1 -- 172.21.15.78:0/3683626672 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558034888d70 con 0x5580349a3110 2024-09-16T01:22:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc989ffb640 10 client.26131 handle_command_reply: tid=0 2024-09-16T01:22:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 client.26131 shutdown 2024-09-16T01:22:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 msgr2=0x55803492fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc99c000c00 tx=0x7fc99c0d0f10 comp rx=0 tx=0).stop 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 shutdown_connections 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5580349a3110 0x5580349a5500 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558034927cb0 0x55803492cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55803491d410 0x558034930200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 --2- 172.21.15.78:0/3683626672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55803491caf0 0x55803492fcc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.703+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 >> 172.21.15.78:0/3683626672 conn(0x558034919060 msgr2=0x55803491e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc989ffb640 0 client.26131 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 shutdown_connections 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9ab28e740 1 -- 172.21.15.78:0/3683626672 wait complete. 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9ab28e740 20 client.26131 trim_cache size 0 max 16384 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:22:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:22:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:22:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:22:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 msgr2=0x7fc98c0a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 0x7fc98c0a5b00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55803491a3e0 tx=0x7fc990004820 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 msgr2=0x7fc9a011a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.704+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 0x7fc9a011a7e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00e9f0 tx=0x7fc99c00eeb0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 shutdown_connections 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fc98c0a3640 0x7fc98c0a5b00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9a011b140 0x7fc9a0116e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9a01095d0 0x7fc9a01168c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 --2- 172.21.15.78:0/858676220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9a01091f0 0x7fc9a011a7e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 >> 172.21.15.78:0/858676220 conn(0x7fc9a006ce00 msgr2=0x7fc9a0070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 shutdown_connections 2024-09-16T01:22:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:22:48.705+0000 7fc9a84f1640 1 -- 172.21.15.78:0/858676220 wait complete. 2024-09-16T01:22:48.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:48 smithi148 ceph-mon[35030]: pgmap v375: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.0 MiB/s wr, 672 op/s 2024-09-16T01:22:49.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:48 smithi078 ceph-mon[30937]: pgmap v375: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.0 MiB/s wr, 672 op/s 2024-09-16T01:22:49.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:48 smithi193 ceph-mon[35701]: pgmap v375: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.0 MiB/s wr, 672 op/s 2024-09-16T01:22:50.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 695.0 s, 585.2 tps, lat 169.630 ms stddev 216.053 2024-09-16T01:22:51.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:50 smithi078 ceph-mon[30937]: pgmap v376: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 588 op/s 2024-09-16T01:22:51.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:50 smithi193 ceph-mon[35701]: pgmap v376: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 588 op/s 2024-09-16T01:22:51.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:50 smithi148 ceph-mon[35030]: pgmap v376: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.3 MiB/s wr, 588 op/s 2024-09-16T01:22:53.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:52 smithi078 ceph-mon[30937]: pgmap v377: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.9 MiB/s wr, 652 op/s 2024-09-16T01:22:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:52 smithi193 ceph-mon[35701]: pgmap v377: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.9 MiB/s wr, 652 op/s 2024-09-16T01:22:53.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:52 smithi148 ceph-mon[35030]: pgmap v377: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.9 MiB/s wr, 652 op/s 2024-09-16T01:22:55.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:54 smithi078 ceph-mon[30937]: pgmap v378: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.8 MiB/s wr, 576 op/s 2024-09-16T01:22:55.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:54 smithi193 ceph-mon[35701]: pgmap v378: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.8 MiB/s wr, 576 op/s 2024-09-16T01:22:55.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:54 smithi148 ceph-mon[35030]: pgmap v378: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.8 MiB/s wr, 576 op/s 2024-09-16T01:22:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 700.0 s, 901.8 tps, lat 112.498 ms stddev 137.535 2024-09-16T01:22:57.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:56 smithi078 ceph-mon[30937]: pgmap v379: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.7 MiB/s wr, 607 op/s 2024-09-16T01:22:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:56 smithi193 ceph-mon[35701]: pgmap v379: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.7 MiB/s wr, 607 op/s 2024-09-16T01:22:57.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:56 smithi148 ceph-mon[35030]: pgmap v379: 129 pgs: 129 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.7 MiB/s wr, 607 op/s 2024-09-16T01:22:59.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:22:58 smithi078 ceph-mon[30937]: pgmap v380: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 600 op/s 2024-09-16T01:22:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:22:58 smithi193 ceph-mon[35701]: pgmap v380: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 600 op/s 2024-09-16T01:22:59.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:22:58 smithi148 ceph-mon[35030]: pgmap v380: 129 pgs: 129 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.0 MiB/s wr, 600 op/s 2024-09-16T01:23:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 705.0 s, 843.4 tps, lat 118.001 ms stddev 130.379 2024-09-16T01:23:00.937 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:00 smithi148 ceph-mon[35030]: pgmap v381: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 507 op/s 2024-09-16T01:23:01.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:00 smithi078 ceph-mon[30937]: pgmap v381: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 507 op/s 2024-09-16T01:23:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:00 smithi193 ceph-mon[35701]: pgmap v381: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.7 MiB/s wr, 507 op/s 2024-09-16T01:23:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.855+0000 7fbe58b86640 1 -- 172.21.15.78:0/1014742727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54100a90 msgr2=0x7fbe54100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.855+0000 7fbe58b86640 1 --2- 172.21.15.78:0/1014742727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54100a90 0x7fbe54100e70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009860 tx=0x7fbe4402a360 comp rx=0 tx=0).stop 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.857+0000 7fbe58b86640 1 -- 172.21.15.78:0/1014742727 shutdown_connections 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.857+0000 7fbe58b86640 1 --2- 172.21.15.78:0/1014742727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54105f10 0x7fbe541063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.857+0000 7fbe58b86640 1 --2- 172.21.15.78:0/1014742727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe541013b0 0x7fbe54105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.857+0000 7fbe58b86640 1 --2- 172.21.15.78:0/1014742727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54100a90 0x7fbe54100e70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.857+0000 7fbe58b86640 1 -- 172.21.15.78:0/1014742727 >> 172.21.15.78:0/1014742727 conn(0x7fbe540fc6b0 msgr2=0x7fbe540fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.858+0000 7fbe58b86640 1 -- 172.21.15.78:0/1014742727 shutdown_connections 2024-09-16T01:23:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.858+0000 7fbe58b86640 1 -- 172.21.15.78:0/1014742727 wait complete. 2024-09-16T01:23:01.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.859+0000 7fbe58b86640 1 Processor -- start 2024-09-16T01:23:01.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 -- start start 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 0x7fbe541a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 0x7fbe541a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 0x7fbe5419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbe54118570 con 0x7fbe541013b0 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbe541183f0 con 0x7fbe54105f10 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.860+0000 7fbe58b86640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe541186f0 con 0x7fbe54100a90 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.861+0000 7fbe52d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 0x7fbe5419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.861+0000 7fbe52d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 0x7fbe5419e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45484/0 (socket says 172.21.15.78:45484) 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.861+0000 7fbe52d9d640 1 -- 172.21.15.78:0/324866159 learned_addr learned my addr 172.21.15.78:0/324866159 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:01.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.861+0000 7fbe5259c640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 0x7fbe541a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 0x7fbe541a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe5259c640 1 -- 172.21.15.78:0/324866159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 msgr2=0x7fbe5419e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe5259c640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 0x7fbe5419e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe5259c640 1 -- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 msgr2=0x7fbe541a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe5259c640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 0x7fbe541a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe5259c640 1 -- 172.21.15.78:0/324866159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5419ef70 con 0x7fbe54100a90 2024-09-16T01:23:01.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.864+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 0x7fbe541a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.866+0000 7fbe5259c640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 0x7fbe541a4630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbe4402a950 tx=0x7fbe44033ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.868+0000 7fbe4b7fe640 1 -- 172.21.15.78:0/324866159 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4403f020 con 0x7fbe54100a90 2024-09-16T01:23:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.868+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5419f200 con 0x7fbe54100a90 2024-09-16T01:23:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.868+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe54106e40 con 0x7fbe54100a90 2024-09-16T01:23:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.869+0000 7fbe4b7fe640 1 -- 172.21.15.78:0/324866159 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe440307f0 con 0x7fbe54100a90 2024-09-16T01:23:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.870+0000 7fbe4b7fe640 1 -- 172.21.15.78:0/324866159 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44037040 con 0x7fbe54100a90 2024-09-16T01:23:01.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.873+0000 7fbe4b7fe640 1 -- 172.21.15.78:0/324866159 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fbe44042020 con 0x7fbe54100a90 2024-09-16T01:23:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.874+0000 7fbe4b7fe640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 0x7fbe340a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.874+0000 7fbe4b7fe640 1 -- 172.21.15.78:0/324866159 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbe440ef780 con 0x7fbe54100a90 2024-09-16T01:23:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.877+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 0x7fbe340a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.878+0000 7fbe5b9f9740 1 Processor -- start 2024-09-16T01:23:01.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.882+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 0x7fbe340a5850 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c006fd0 tx=0x7fbe3c008040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.882+0000 7fbe5b9f9740 1 -- start start 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.882+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa06af0 0x5575caa06ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.882+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa07410 0x5575caa11770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.882+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.883+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5575ca96b2f0 con 0x5575caa11cb0 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.883+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5575ca913270 con 0x5575caa07410 2024-09-16T01:23:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.883+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575ca9672b0 con 0x5575caa06af0 2024-09-16T01:23:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.883+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa07410 0x5575caa11770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.884+0000 7fbe52d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.884+0000 7fbe52d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35910/0 (socket says 172.21.15.78:35910) 2024-09-16T01:23:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.884+0000 7fbe52d9d640 1 -- 172.21.15.78:0/1926842232 learned_addr learned my addr 172.21.15.78:0/1926842232 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe5259c640 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa06af0 0x5575caa06ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 -- 172.21.15.78:0/1926842232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa06af0 msgr2=0x5575caa06ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa06af0 0x5575caa06ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 -- 172.21.15.78:0/1926842232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa07410 msgr2=0x5575caa11770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa07410 0x5575caa11770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 -- 172.21.15.78:0/1926842232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575ca912340 con 0x5575caa11cb0 2024-09-16T01:23:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.885+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00e610 tx=0x7fbe4c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11b11c78beefd06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.886+0000 7fbe48ff9640 1 -- 172.21.15.78:0/1926842232 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c003970 con 0x5575caa11cb0 2024-09-16T01:23:01.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.886+0000 7fbe48ff9640 1 -- 172.21.15.78:0/1926842232 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe4c00d890 con 0x5575caa11cb0 2024-09-16T01:23:01.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.887+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/1926842232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 msgr2=0x5575caa140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.887+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00e610 tx=0x7fbe4c0095c0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.890+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/1926842232 shutdown_connections 2024-09-16T01:23:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.890+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa140a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.890+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa07410 0x5575caa11770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.890+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/1926842232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa06af0 0x5575caa06ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.890+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/1926842232 >> 172.21.15.78:0/1926842232 conn(0x5575caa03060 msgr2=0x5575caa03440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:01.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.898+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/1926842232 shutdown_connections 2024-09-16T01:23:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.904+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/1926842232 wait complete. 2024-09-16T01:23:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.904+0000 7fbe5b9f9740 1 -- start start 2024-09-16T01:23:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.905+0000 7fbe5b9f9740 1 Processor -- start 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa06af0 0x5575caa0ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa0f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5575caa148d0 con 0x5575caa11cb0 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5575ca8f4290 con 0x5575caa06af0 2024-09-16T01:23:01.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575caa14a50 con 0x5575caa07410 2024-09-16T01:23:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.906+0000 7fbe5b9f9740 10 client.0 _ll_register_callbacks cb 0x5575ca925140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.907+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.907+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:40604/0 (socket says 172.21.15.78:40604) 2024-09-16T01:23:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.907+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2138303233 learned_addr learned my addr 172.21.15.78:0/2138303233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.907+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa0f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.908+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa06af0 msgr2=0x5575caa0ea20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:23:01.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.908+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa06af0 0x5575caa0ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.908+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2138303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 msgr2=0x5575caa0f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.908+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa0f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.908+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2138303233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575ca67e210 con 0x5575caa07410 2024-09-16T01:23:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.910+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00cea0 tx=0x7fbe3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df82c9cf0916fa3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.912+0000 7fbe41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:23:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.913+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0115e0 con 0x5575caa07410 2024-09-16T01:23:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.913+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbe3c011780 con 0x5575caa07410 2024-09-16T01:23:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.913+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe4c00d890 con 0x5575caa07410 2024-09-16T01:23:01.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.916+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbe3c011980 con 0x5575caa07410 2024-09-16T01:23:01.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.916+0000 7fbe5b9f9740 10 client.26140 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:01.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.916+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575ca74a620 con 0x5575caa07410 2024-09-16T01:23:01.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe4c00d890 con 0x5575caa07410 2024-09-16T01:23:01.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbe3c01fb80 con 0x5575caa07410 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 10 client.26140 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 10 client.26140 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 20 client.26140 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa33120 0x5575caa355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 4 client.26140 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.917+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575ca978a20 con 0x5575caa33120 2024-09-16T01:23:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.918+0000 7fbe5259c640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa33120 0x5575caa355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.923+0000 7fbe5259c640 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa33120 0x5575caa355c0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688990891cbb9613 server_cookie=4965ca01cdcc7ad1 in_seq=0 out_seq=0 2024-09-16T01:23:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.925+0000 7fbe41ffb640 10 client.26140 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.929+0000 7fbe41ffb640 1 -- 172.21.15.78:0/2138303233 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5575ca978a20 con 0x5575caa33120 2024-09-16T01:23:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.929+0000 7fbe41ffb640 10 client.26140 handle_command_reply: tid=0 2024-09-16T01:23:01.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.929+0000 7fbe5b9f9740 1 client.26140 shutdown 2024-09-16T01:23:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 msgr2=0x5575caa0ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00cea0 tx=0x7fbe3c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 shutdown_connections 2024-09-16T01:23:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa33120 0x5575caa355c0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa11cb0 0x5575caa0f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa07410 0x5575caa0ef60 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00cea0 tx=0x7fbe3c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/2138303233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa06af0 0x5575caa0ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 >> 172.21.15.78:0/2138303233 conn(0x5575caa03060 msgr2=0x5575caa08d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.931+0000 7fbe41ffb640 0 client.26140 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.934+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 shutdown_connections 2024-09-16T01:23:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.934+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/2138303233 wait complete. 2024-09-16T01:23:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.935+0000 7fbe5b9f9740 20 client.26140 trim_cache size 0 max 16384 2024-09-16T01:23:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.969+0000 7fbe5b9f9740 1 Processor -- start 2024-09-16T01:23:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 -- start start 2024-09-16T01:23:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa10990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35918/0 (socket says 172.21.15.78:35918) 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5259c640 1 -- 172.21.15.78:0/4268266296 learned_addr learned my addr 172.21.15.78:0/4268266296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa3ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38012a30 con 0x5575caa06af0 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38010300 con 0x5575caa11cb0 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.970+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800e170 con 0x5575caa0db10 2024-09-16T01:23:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.971+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa3ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 -- 172.21.15.78:0/4268266296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 msgr2=0x5575caa3ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa3ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 -- 172.21.15.78:0/4268266296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 msgr2=0x5575caa10990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa10990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 -- 172.21.15.78:0/4268266296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575caa07240 con 0x5575caa06af0 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.972+0000 7fbe5259c640 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbe440bb480 tx=0x7fbe44005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb868b68ae60424 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.973+0000 7fbe43fff640 1 -- 172.21.15.78:0/4268266296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440d4070 con 0x5575caa06af0 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.973+0000 7fbe43fff640 1 -- 172.21.15.78:0/4268266296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe440bfb30 con 0x5575caa06af0 2024-09-16T01:23:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.973+0000 7fbe43fff640 1 -- 172.21.15.78:0/4268266296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440d4070 con 0x5575caa06af0 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.974+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 msgr2=0x5575caa0f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.974+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbe440bb480 tx=0x7fbe44005f90 comp rx=0 tx=0).stop 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.974+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 shutdown_connections 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.974+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa3ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.975+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa10990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.975+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/4268266296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0f2d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.975+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 >> 172.21.15.78:0/4268266296 conn(0x5575caa03060 msgr2=0x5575caa10280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.977+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 shutdown_connections 2024-09-16T01:23:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.977+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/4268266296 wait complete. 2024-09-16T01:23:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.978+0000 7fbe5b9f9740 1 -- start start 2024-09-16T01:23:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.978+0000 7fbe5b9f9740 1 Processor -- start 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.978+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa19dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35934/0 (socket says 172.21.15.78:35934) 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5259c640 1 -- 172.21.15.78:0/3728068554 learned_addr learned my addr 172.21.15.78:0/3728068554 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa1a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800e7c0 con 0x5575caa06af0 2024-09-16T01:23:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800fc30 con 0x5575caa11cb0 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa19dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa1a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800ee90 con 0x5575caa0db10 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.979+0000 7fbe5b9f9740 10 client.0 _ll_register_callbacks cb 0x7fbe3c00af00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 -- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 msgr2=0x5575caa1a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa1a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 -- 172.21.15.78:0/3728068554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 msgr2=0x5575caa0c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575caa12d80 con 0x5575caa0db10 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe52d9d640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa1a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe5259c640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa19dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5575caa0fb10 tx=0x7fbe3c00a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27f837531210234a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:23:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c011b20 con 0x5575caa0db10 2024-09-16T01:23:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.980+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9525+0+0 (secure 0 0 0) 0x7fbe3c011cc0 con 0x5575caa0db10 2024-09-16T01:23:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.981+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe440bfb30 con 0x5575caa0db10 2024-09-16T01:23:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.981+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbe3c011ec0 con 0x5575caa0db10 2024-09-16T01:23:01.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe5b9f9740 10 client.26146 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575caa9e0a0 con 0x5575caa0db10 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c024070 con 0x5575caa0db10 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbe3c01e6a0 con 0x5575caa0db10 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe5b9f9740 10 client.26146 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe5b9f9740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.983+0000 7fbe5b9f9740 20 client.26146 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.984+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa9e7b0 0x5575caaa0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.984+0000 7fbe5b9f9740 4 client.26146 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:23:01.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.984+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fbe440bfb30 con 0x5575caa9e7b0 2024-09-16T01:23:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.986+0000 7fbe5259c640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa9e7b0 0x5575caaa0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.986+0000 7fbe5259c640 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa9e7b0 0x5575caaa0ba0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c714c467ac85191d server_cookie=af8c11df6ba7994 in_seq=0 out_seq=0 2024-09-16T01:23:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:01.986+0000 7fbe41ffb640 10 client.26146 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.871+0000 7fbe41ffb640 1 -- 172.21.15.78:0/3728068554 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fbe440bfb30 con 0x5575caa9e7b0 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.871+0000 7fbe41ffb640 10 client.26146 handle_command_reply: tid=0 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.871+0000 7fbe5b9f9740 1 client.26146 shutdown 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 msgr2=0x5575caa19dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa19dc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5575caa0fb10 tx=0x7fbe3c00a140 comp rx=0 tx=0).stop 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 shutdown_connections 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575caa9e7b0 0x5575caaa0ba0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575caa11cb0 0x5575caa1a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575caa0db10 0x5575caa19dc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 --2- 172.21.15.78:0/3728068554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575caa06af0 0x5575caa0c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 >> 172.21.15.78:0/3728068554 conn(0x5575caa03060 msgr2=0x5575caa08d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe41ffb640 0 client.26146 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 shutdown_connections 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.872+0000 7fbe5b9f9740 1 -- 172.21.15.78:0/3728068554 wait complete. 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe5b9f9740 20 client.26146 trim_cache size 0 max 16384 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:23:02.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:23:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:23:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:23:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 msgr2=0x7fbe340a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:02.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 0x7fbe340a5850 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c006fd0 tx=0x7fbe3c008040 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 msgr2=0x7fbe541a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 0x7fbe541a4630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbe4402a950 tx=0x7fbe44033ae0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.873+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 shutdown_connections 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fbe340a3390 0x7fbe340a5850 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe54105f10 0x7fbe5419e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe541013b0 0x7fbe541a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 --2- 172.21.15.78:0/324866159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe54100a90 0x7fbe541a4630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 >> 172.21.15.78:0/324866159 conn(0x7fbe540fc6b0 msgr2=0x7fbe541037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 shutdown_connections 2024-09-16T01:23:02.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:02.874+0000 7fbe58b86640 1 -- 172.21.15.78:0/324866159 wait complete. 2024-09-16T01:23:03.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:02 smithi078 ceph-mon[30937]: pgmap v382: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 599 op/s 2024-09-16T01:23:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:02 smithi193 ceph-mon[35701]: pgmap v382: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 599 op/s 2024-09-16T01:23:03.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:02 smithi148 ceph-mon[35030]: pgmap v382: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 599 op/s 2024-09-16T01:23:05.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:04 smithi078 ceph-mon[30937]: pgmap v383: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 579 op/s 2024-09-16T01:23:05.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:04 smithi078 ceph-mon[30937]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T01:23:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:04 smithi193 ceph-mon[35701]: pgmap v383: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 579 op/s 2024-09-16T01:23:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:04 smithi193 ceph-mon[35701]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T01:23:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:04 smithi148 ceph-mon[35030]: pgmap v383: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.2 MiB/s wr, 579 op/s 2024-09-16T01:23:05.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:04 smithi148 ceph-mon[35030]: osdmap e107: 12 total, 12 up, 12 in 2024-09-16T01:23:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 710.0 s, 493.4 tps, lat 206.941 ms stddev 331.716 2024-09-16T01:23:07.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:06 smithi078 ceph-mon[30937]: pgmap v385: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.7 MiB/s wr, 756 op/s 2024-09-16T01:23:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:06 smithi193 ceph-mon[35701]: pgmap v385: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.7 MiB/s wr, 756 op/s 2024-09-16T01:23:07.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:06 smithi148 ceph-mon[35030]: pgmap v385: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.7 MiB/s wr, 756 op/s 2024-09-16T01:23:09.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:08 smithi078 ceph-mon[30937]: pgmap v386: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.5 MiB/s wr, 810 op/s 2024-09-16T01:23:09.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:08 smithi193 ceph-mon[35701]: pgmap v386: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.5 MiB/s wr, 810 op/s 2024-09-16T01:23:09.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:08 smithi148 ceph-mon[35030]: pgmap v386: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.5 MiB/s wr, 810 op/s 2024-09-16T01:23:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 715.0 s, 644.0 tps, lat 146.482 ms stddev 182.809 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: pgmap v387: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.8 MiB/s wr, 860 op/s 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:10 smithi078 ceph-mon[30937]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T01:23:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: pgmap v387: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.8 MiB/s wr, 860 op/s 2024-09-16T01:23:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:23:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:10 smithi193 ceph-mon[35701]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: pgmap v387: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 9.8 MiB/s wr, 860 op/s 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:11.137 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:10 smithi148 ceph-mon[35030]: osdmap e108: 12 total, 12 up, 12 in 2024-09-16T01:23:12.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:11 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:23:12.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:11 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:23:12.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:11 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:12.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:11 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:23:12.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:11 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:23:12.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:11 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:12.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:11 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:23:12.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:11 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:23:12.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:11 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:23:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:12 smithi078 ceph-mon[30937]: pgmap v389: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-09-16T01:23:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:12 smithi193 ceph-mon[35701]: pgmap v389: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-09-16T01:23:13.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:12 smithi148 ceph-mon[35030]: pgmap v389: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.7 MiB/s wr, 883 op/s 2024-09-16T01:23:15.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:14 smithi078 ceph-mon[30937]: pgmap v390: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.0 MiB/s wr, 788 op/s 2024-09-16T01:23:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:14 smithi193 ceph-mon[35701]: pgmap v390: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.0 MiB/s wr, 788 op/s 2024-09-16T01:23:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:14 smithi148 ceph-mon[35030]: pgmap v390: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.0 MiB/s wr, 788 op/s 2024-09-16T01:23:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 720.0 s, 634.6 tps, lat 163.606 ms stddev 199.870 2024-09-16T01:23:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.035+0000 7f195cc2e640 1 -- 172.21.15.78:0/2796414078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 msgr2=0x7f19581122c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.035+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2796414078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 0x7f19581122c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f194c009900 tx=0x7f194c02a4d0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.042+0000 7f195cc2e640 1 -- 172.21.15.78:0/2796414078 shutdown_connections 2024-09-16T01:23:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.042+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2796414078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1958112800 0x7f1958114bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.042+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2796414078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 0x7f19581122c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.042+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2796414078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19580ffb70 0x7f19580fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.042+0000 7f195cc2e640 1 -- 172.21.15.78:0/2796414078 >> 172.21.15.78:0/2796414078 conn(0x7f19580fba30 msgr2=0x7f19580fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.050+0000 7f195cc2e640 1 -- 172.21.15.78:0/2796414078 shutdown_connections 2024-09-16T01:23:16.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.054+0000 7f195cc2e640 1 -- 172.21.15.78:0/2796414078 wait complete. 2024-09-16T01:23:16.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.054+0000 7f195cc2e640 1 Processor -- start 2024-09-16T01:23:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 -- start start 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 0x7f1958112290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958112800 0x7f195810af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f19581184e0 con 0x7f1958100490 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1958118360 con 0x7f19580ffb70 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195cc2e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1958118660 con 0x7f1958112800 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45968/0 (socket says 172.21.15.78:45968) 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.057+0000 7f195659c640 1 -- 172.21.15.78:0/2332721809 learned_addr learned my addr 172.21.15.78:0/2332721809 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 -- 172.21.15.78:0/2332721809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958112800 msgr2=0x7f195810af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958112800 0x7f195810af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 -- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 msgr2=0x7f1958112290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:23:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 0x7f1958112290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 -- 172.21.15.78:0/2332721809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195810b840 con 0x7f19580ffb70 2024-09-16T01:23:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.058+0000 7f195659c640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f194800c890 tx=0x7f194800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.059+0000 7f19477fe640 1 -- 172.21.15.78:0/2332721809 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948004220 con 0x7f19580ffb70 2024-09-16T01:23:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.059+0000 7f19477fe640 1 -- 172.21.15.78:0/2332721809 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19480043c0 con 0x7f19580ffb70 2024-09-16T01:23:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.059+0000 7f19477fe640 1 -- 172.21.15.78:0/2332721809 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19480049e0 con 0x7f19580ffb70 2024-09-16T01:23:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.059+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195810bb30 con 0x7f19580ffb70 2024-09-16T01:23:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.059+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195810fb10 con 0x7f19580ffb70 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.068+0000 7f195f9cd740 1 Processor -- start 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.068+0000 7f195f9cd740 1 -- start start 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb53770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5654ceaad2f0 con 0x5654ceb49410 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5654cea55270 con 0x5654ceb48af0 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f195f9cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5654ceaa92b0 con 0x5654ceb53cb0 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45982/0 (socket says 172.21.15.78:45982) 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 -- 172.21.15.78:0/150910601 learned_addr learned my addr 172.21.15.78:0/150910601 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 -- 172.21.15.78:0/150910601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 msgr2=0x5654ceb560a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb560a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 -- 172.21.15.78:0/150910601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 msgr2=0x5654ceb53770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb53770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.069+0000 7f1956d9d640 1 -- 172.21.15.78:0/150910601 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654cea54340 con 0x5654ceb48af0 2024-09-16T01:23:16.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.070+0000 7f1956d9d640 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d10 tx=0x7f195000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fed7b2940ae4616 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.072+0000 7f19477fe640 1 -- 172.21.15.78:0/2332721809 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f19480079b0 con 0x7f19580ffb70 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.072+0000 7f1944ff9640 1 -- 172.21.15.78:0/150910601 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950016070 con 0x5654ceb48af0 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.072+0000 7f1944ff9640 1 -- 172.21.15.78:0/150910601 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950005020 con 0x5654ceb48af0 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.072+0000 7f1944ff9640 1 -- 172.21.15.78:0/150910601 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500116b0 con 0x5654ceb48af0 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.073+0000 7f19477fe640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 0x7f19400a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.073+0000 7f19477fe640 1 -- 172.21.15.78:0/2332721809 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f19480d00c0 con 0x7f19580ffb70 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 -- 172.21.15.78:0/150910601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 msgr2=0x5654ceb48ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1950009d10 tx=0x7f195000e6c0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 -- 172.21.15.78:0/150910601 shutdown_connections 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb53770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 --2- 172.21.15.78:0/150910601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb48ed0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 -- 172.21.15.78:0/150910601 >> 172.21.15.78:0/150910601 conn(0x5654ceb45060 msgr2=0x5654ceb45440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.074+0000 7f195f9cd740 1 -- 172.21.15.78:0/150910601 shutdown_connections 2024-09-16T01:23:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.075+0000 7f195659c640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 0x7f19400a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.076+0000 7f195f9cd740 1 -- 172.21.15.78:0/150910601 wait complete. 2024-09-16T01:23:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.076+0000 7f195659c640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 0x7f19400a5b50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x5654ceb45ec0 tx=0x7f1948000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.076+0000 7f195f9cd740 1 -- start start 2024-09-16T01:23:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.076+0000 7f195f9cd740 1 Processor -- start 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb50b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb51600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5654ceb56880 con 0x5654ceb49410 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5654cea36290 con 0x5654ceb48af0 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5654ceb56a00 con 0x5654ceb53cb0 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f195f9cd740 10 client.0 _ll_register_callbacks cb 0x5654cea67140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58678/0 (socket says 172.21.15.78:58678) 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 -- 172.21.15.78:0/3726309086 learned_addr learned my addr 172.21.15.78:0/3726309086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 -- 172.21.15.78:0/3726309086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 msgr2=0x5654ceb51600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb51600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 -- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 msgr2=0x5654ceb50b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb50b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.081+0000 7f1955d9b640 1 -- 172.21.15.78:0/3726309086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ce7c0210 con 0x5654ceb49410 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f1955d9b640 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f194c000c00 tx=0x7f194c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54170b60997d083f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f193dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c03e030 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f194c003800 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950005020 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f194c04b050 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f195f9cd740 10 client.16776 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.082+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5654ce88c620 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950005020 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f194c031a10 con 0x5654ceb49410 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 10 client.16776 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 10 client.16776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 20 client.16776 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654ceb641a0 0x5654ceb66640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 4 client.16776 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654ceabaa20 con 0x5654ceb641a0 2024-09-16T01:23:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.083+0000 7f1956d9d640 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654ceb641a0 0x5654ceb66640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.084+0000 7f193dffb640 10 client.16776 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.084+0000 7f1956d9d640 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654ceb641a0 0x5654ceb66640 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d676ff74e322234c server_cookie=33b400250e1bb207 in_seq=0 out_seq=0 2024-09-16T01:23:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f193dffb640 1 -- 172.21.15.78:0/3726309086 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5654ceabaa20 con 0x5654ceb641a0 2024-09-16T01:23:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f193dffb640 10 client.16776 handle_command_reply: tid=0 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 client.16776 shutdown 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 msgr2=0x5654ceb510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f194c000c00 tx=0x7f194c004270 comp rx=0 tx=0).stop 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 shutdown_connections 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654ceb641a0 0x5654ceb66640 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb51600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb49410 0x5654ceb510c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 --2- 172.21.15.78:0/3726309086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb48af0 0x5654ceb50b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 >> 172.21.15.78:0/3726309086 conn(0x5654ceb45060 msgr2=0x5654ceb4adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f193dffb640 0 client.16776 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 shutdown_connections 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 1 -- 172.21.15.78:0/3726309086 wait complete. 2024-09-16T01:23:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.085+0000 7f195f9cd740 20 client.16776 trim_cache size 0 max 16384 2024-09-16T01:23:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.103+0000 7f195f9cd740 1 Processor -- start 2024-09-16T01:23:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.103+0000 7f195f9cd740 1 -- start start 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb49410 0x5654ceb51430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb53cb0 0x5654ceb52af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1938012870 con 0x5654ceb48af0 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1938010140 con 0x5654ceb53cb0 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f195f9cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f193800dfb0 con 0x5654ceb49410 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f1956d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f1956d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58682/0 (socket says 172.21.15.78:58682) 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f1956d9d640 1 -- 172.21.15.78:0/1356448784 learned_addr learned my addr 172.21.15.78:0/1356448784 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f1955d9b640 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb49410 0x5654ceb51430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.104+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb53cb0 0x5654ceb52af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 -- 172.21.15.78:0/1356448784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb49410 msgr2=0x5654ceb51430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb49410 0x5654ceb51430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 -- 172.21.15.78:0/1356448784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb53cb0 msgr2=0x5654ceb52af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb53cb0 0x5654ceb52af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 -- 172.21.15.78:0/1356448784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654ceb4f0b0 con 0x5654ceb48af0 2024-09-16T01:23:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f195000e700 tx=0x7f1950005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f138a5c7b73b5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f193ffff640 1 -- 172.21.15.78:0/1356448784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950016040 con 0x5654ceb48af0 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f193ffff640 1 -- 172.21.15.78:0/1356448784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950011950 con 0x5654ceb48af0 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.105+0000 7f193ffff640 1 -- 172.21.15.78:0/1356448784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950027920 con 0x5654ceb48af0 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 -- 172.21.15.78:0/1356448784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 msgr2=0x5654ceb4b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f195000e700 tx=0x7f1950005a10 comp rx=0 tx=0).stop 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 -- 172.21.15.78:0/1356448784 shutdown_connections 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb53cb0 0x5654ceb52af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb49410 0x5654ceb51430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1356448784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb4b560 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 -- 172.21.15.78:0/1356448784 >> 172.21.15.78:0/1356448784 conn(0x5654ceb45060 msgr2=0x5654ceb523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 -- 172.21.15.78:0/1356448784 shutdown_connections 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.106+0000 7f195f9cd740 1 -- 172.21.15.78:0/1356448784 wait complete. 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 -- start start 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 Processor -- start 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb5c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb4dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f193800e600 con 0x5654ceb48af0 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f193800fa70 con 0x5654ceb49410 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f193800ecd0 con 0x5654ceb53cb0 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f1956d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb5c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f1955d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f195f9cd740 10 client.0 _ll_register_callbacks cb 0x7f194c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f1955d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:45988/0 (socket says 172.21.15.78:45988) 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f1955d9b640 1 -- 172.21.15.78:0/1035810040 learned_addr learned my addr 172.21.15.78:0/1035810040 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.107+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb4dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 -- 172.21.15.78:0/1035810040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 msgr2=0x5654ceb4dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb4dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 -- 172.21.15.78:0/1035810040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 msgr2=0x5654ceb5c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb5c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 -- 172.21.15.78:0/1035810040 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ceb4dff0 con 0x5654ceb49410 2024-09-16T01:23:16.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb4dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f1955d9b640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f194c033000 tx=0x7f194c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a7e90c86cb16e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f193dffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c03e040 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1950011950 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f194c03ab30 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f194c035040 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f195f9cd740 10 client.26084 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5654cebcec00 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.108+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950011950 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f194c057030 con 0x5654ceb49410 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 10 client.26084 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 10 client.26084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 20 client.26084 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654cebcf400 0x5654cebd1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 4 client.26084 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:23:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654ceab4d70 con 0x5654cebcf400 2024-09-16T01:23:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.109+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654cebcf400 0x5654cebd1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.110+0000 7f193dffb640 10 client.26084 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.110+0000 7f1956d9d640 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654cebcf400 0x5654cebd1820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e774dabdb649ec server_cookie=9f21e2c7d6da5496 in_seq=0 out_seq=0 2024-09-16T01:23:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.256+0000 7f193dffb640 1 -- 172.21.15.78:0/1035810040 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654ceab4d70 con 0x5654cebcf400 2024-09-16T01:23:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.256+0000 7f193dffb640 10 client.26084 handle_command_reply: tid=0 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.256+0000 7f195f9cd740 1 client.26084 shutdown 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.256+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 msgr2=0x5654ceb4d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.256+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f194c033000 tx=0x7f194c0045e0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 shutdown_connections 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5654cebcf400 0x5654cebd1820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f193dffb640 0 client.26084 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654ceb53cb0 0x5654ceb4dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654ceb49410 0x5654ceb4d570 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 --2- 172.21.15.78:0/1035810040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654ceb48af0 0x5654ceb5c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 >> 172.21.15.78:0/1035810040 conn(0x5654ceb45060 msgr2=0x5654ceb441c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 shutdown_connections 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 1 -- 172.21.15.78:0/1035810040 wait complete. 2024-09-16T01:23:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.257+0000 7f195f9cd740 20 client.26084 trim_cache size 0 max 16384 2024-09-16T01:23:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:23:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:23:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:23:16.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:23:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 msgr2=0x7f19400a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 0x7f19400a5b50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x5654ceb45ec0 tx=0x7f1948000bc0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 msgr2=0x7f1958111d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f194800c890 tx=0x7f194800cd50 comp rx=0 tx=0).stop 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 shutdown_connections 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f19400a3690 0x7f19400a5b50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958112800 0x7f195810af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1958100490 0x7f1958112290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 --2- 172.21.15.78:0/2332721809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19580ffb70 0x7f1958111d50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 >> 172.21.15.78:0/2332721809 conn(0x7f19580fba30 msgr2=0x7f19580fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 shutdown_connections 2024-09-16T01:23:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:16.258+0000 7f195cc2e640 1 -- 172.21.15.78:0/2332721809 wait complete. 2024-09-16T01:23:16.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:16 smithi148 ceph-mon[35030]: pgmap v391: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.3 MiB/s wr, 676 op/s 2024-09-16T01:23:16.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:16 smithi148 ceph-mon[35030]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T01:23:17.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:16 smithi078 ceph-mon[30937]: pgmap v391: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.3 MiB/s wr, 676 op/s 2024-09-16T01:23:17.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:16 smithi078 ceph-mon[30937]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T01:23:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:16 smithi193 ceph-mon[35701]: pgmap v391: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.3 MiB/s wr, 676 op/s 2024-09-16T01:23:17.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:16 smithi193 ceph-mon[35701]: osdmap e109: 12 total, 12 up, 12 in 2024-09-16T01:23:18.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:18 smithi148 ceph-mon[35030]: pgmap v393: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.0 MiB/s wr, 592 op/s 2024-09-16T01:23:19.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:18 smithi078 ceph-mon[30937]: pgmap v393: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.0 MiB/s wr, 592 op/s 2024-09-16T01:23:19.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:18 smithi193 ceph-mon[35701]: pgmap v393: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.0 MiB/s wr, 592 op/s 2024-09-16T01:23:20.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:20 smithi148 ceph-mon[35030]: pgmap v394: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-09-16T01:23:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 725.0 s, 581.8 tps, lat 172.974 ms stddev 213.672 2024-09-16T01:23:21.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:20 smithi078 ceph-mon[30937]: pgmap v394: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-09-16T01:23:21.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:20 smithi193 ceph-mon[35701]: pgmap v394: 129 pgs: 129 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-09-16T01:23:22.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:22 smithi148 ceph-mon[35030]: pgmap v395: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-09-16T01:23:23.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:22 smithi078 ceph-mon[30937]: pgmap v395: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-09-16T01:23:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:22 smithi193 ceph-mon[35701]: pgmap v395: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-09-16T01:23:24.886 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:24 smithi148 ceph-mon[35030]: pgmap v396: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 574 op/s 2024-09-16T01:23:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:24 smithi078 ceph-mon[30937]: pgmap v396: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 574 op/s 2024-09-16T01:23:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:24 smithi193 ceph-mon[35701]: pgmap v396: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 574 op/s 2024-09-16T01:23:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 730.0 s, 561.0 tps, lat 175.914 ms stddev 206.495 2024-09-16T01:23:26.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:26 smithi148 ceph-mon[35030]: pgmap v397: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 576 op/s 2024-09-16T01:23:27.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:26 smithi078 ceph-mon[30937]: pgmap v397: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 576 op/s 2024-09-16T01:23:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:26 smithi193 ceph-mon[35701]: pgmap v397: 129 pgs: 129 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 576 op/s 2024-09-16T01:23:28.885 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:28 smithi148 ceph-mon[35030]: pgmap v398: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 6.8 MiB/s wr, 579 op/s 2024-09-16T01:23:29.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:28 smithi078 ceph-mon[30937]: pgmap v398: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 6.8 MiB/s wr, 579 op/s 2024-09-16T01:23:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:28 smithi193 ceph-mon[35701]: pgmap v398: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 6.8 MiB/s wr, 579 op/s 2024-09-16T01:23:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 -- 172.21.15.78:0/599435167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1011c0 msgr2=0x7f080c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 --2- 172.21.15.78:0/599435167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1011c0 0x7f080c10df10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0800009900 tx=0x7f080002a410 comp rx=0 tx=0).stop 2024-09-16T01:23:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 -- 172.21.15.78:0/599435167 shutdown_connections 2024-09-16T01:23:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 --2- 172.21.15.78:0/599435167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c10e450 0x7f080c110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 --2- 172.21.15.78:0/599435167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1011c0 0x7f080c10df10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 --2- 172.21.15.78:0/599435167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c1008a0 0x7f080c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 -- 172.21.15.78:0/599435167 >> 172.21.15.78:0/599435167 conn(0x7f080c0fc6b0 msgr2=0x7f080c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 -- 172.21.15.78:0/599435167 shutdown_connections 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.376+0000 7f0812e77640 1 -- 172.21.15.78:0/599435167 wait complete. 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 Processor -- start 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 -- start start 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c10e450 0x7f080c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1140e0 con 0x7f080c1008a0 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f080c113f60 con 0x7f080c1011c0 2024-09-16T01:23:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0812e77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f080c114260 con 0x7f080c10e450 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f080bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f080bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:35456/0 (socket says 172.21.15.78:35456) 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f080bfff640 1 -- 172.21.15.78:0/1445684620 learned_addr learned my addr 172.21.15.78:0/1445684620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52744/0 (socket says 172.21.15.78:52744) 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0811414640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c10e450 0x7f080c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 -- 172.21.15.78:0/1445684620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c10e450 msgr2=0x7f080c19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c10e450 0x7f080c19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 -- 172.21.15.78:0/1445684620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 msgr2=0x7f080c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.377+0000 7f0810c13640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0810c13640 1 -- 172.21.15.78:0/1445684620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c19f5d0 con 0x7f080c1008a0 2024-09-16T01:23:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f080bfff640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:23:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0810c13640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00c970 tx=0x7f07fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0809ffb640 1 -- 172.21.15.78:0/1445684620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc007bb0 con 0x7f080c1008a0 2024-09-16T01:23:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0809ffb640 1 -- 172.21.15.78:0/1445684620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07fc007d50 con 0x7f080c1008a0 2024-09-16T01:23:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0809ffb640 1 -- 172.21.15.78:0/1445684620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc00f450 con 0x7f080c1008a0 2024-09-16T01:23:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080c19f8c0 con 0x7f080c1008a0 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.378+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080c1a7150 con 0x7f080c1008a0 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.381+0000 7f0809ffb640 1 -- 172.21.15.78:0/1445684620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f07fc004ef0 con 0x7f080c1008a0 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.381+0000 7f0809ffb640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 0x7f07f40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0809ffb640 1 -- 172.21.15.78:0/1445684620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f07fc0cfcb0 con 0x7f080c1008a0 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f080bfff640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 0x7f07f40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 Processor -- start 2024-09-16T01:23:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 -- start start 2024-09-16T01:23:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5ce52f0 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5c8d270 con 0x55ffa5d80af0 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0815c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5ce12b0 con 0x55ffa5d81410 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0810c13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0810c13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:35460/0 (socket says 172.21.15.78:35460) 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0810c13640 1 -- 172.21.15.78:0/1078349718 learned_addr learned my addr 172.21.15.78:0/1078349718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f0811414640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.382+0000 7f080bfff640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 0x7f07f40a5920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f080c10e870 tx=0x7f0800005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f080bfff640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 -- 172.21.15.78:0/1078349718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 msgr2=0x55ffa5d8b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 -- 172.21.15.78:0/1078349718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d8bcb0 msgr2=0x55ffa5d8e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 -- 172.21.15.78:0/1078349718 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa5c8c340 con 0x55ffa5d80af0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f080bfff640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f0810c13640 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0980e0 tx=0x7f07fc09bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db60ca78f199e37e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f07f2ffd640 1 -- 172.21.15.78:0/1078349718 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc01d040 con 0x55ffa5d80af0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f07f2ffd640 1 -- 172.21.15.78:0/1078349718 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07fc099d20 con 0x55ffa5d80af0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.383+0000 7f07f2ffd640 1 -- 172.21.15.78:0/1078349718 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc098480 con 0x55ffa5d80af0 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- 172.21.15.78:0/1078349718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 msgr2=0x55ffa5d80ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0980e0 tx=0x7f07fc09bdc0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- 172.21.15.78:0/1078349718 shutdown_connections 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 --2- 172.21.15.78:0/1078349718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d80af0 0x55ffa5d80ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- 172.21.15.78:0/1078349718 >> 172.21.15.78:0/1078349718 conn(0x55ffa5d7d060 msgr2=0x55ffa5d7d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- 172.21.15.78:0/1078349718 shutdown_connections 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- 172.21.15.78:0/1078349718 wait complete. 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.384+0000 7f0815c09740 1 -- start start 2024-09-16T01:23:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 Processor -- start 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d890e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d89620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5d8e880 con 0x55ffa5d80af0 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5c6e290 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ffa5d8ea00 con 0x55ffa5d81410 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52754/0 (socket says 172.21.15.78:52754) 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0810c13640 1 -- 172.21.15.78:0/3858286550 learned_addr learned my addr 172.21.15.78:0/3858286550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0811414640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d89620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0815c09740 10 client.0 _ll_register_callbacks cb 0x55ffa5c9f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f080bfff640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d890e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0811414640 1 -- 172.21.15.78:0/3858286550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 msgr2=0x55ffa5d890e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0811414640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d890e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.385+0000 7f0811414640 1 -- 172.21.15.78:0/3858286550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 msgr2=0x55ffa5d88ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0811414640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0811414640 1 -- 172.21.15.78:0/3858286550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa59f8210 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0810c13640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0811414640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d89620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f080c06b430 tx=0x7f0804010eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d3b3f22e48ddbb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080400fce0 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f0804003970 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07fc099d20 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0804024050 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0815c09740 10 client.26093 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ffa5ac4620 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07fc099d20 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.386+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0804019410 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 10 client.26093 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 20 client.26093 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5d9bb90 0x55ffa5d9e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 4 client.26093 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffa5cf2a20 con 0x55ffa5d9bb90 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0810c13640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5d9bb90 0x55ffa5d9e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f0810c13640 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5d9bb90 0x55ffa5d9e000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=570bf7884921c6d8 server_cookie=8fd03bea20dc6eb3 in_seq=0 out_seq=0 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.387+0000 7f07ebfff640 10 client.26093 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f07ebfff640 1 -- 172.21.15.78:0/3858286550 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ffa5cf2a20 con 0x55ffa5d9bb90 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f07ebfff640 10 client.26093 handle_command_reply: tid=0 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 client.26093 shutdown 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 msgr2=0x55ffa5d89620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d89620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f080c06b430 tx=0x7f0804010eb0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 shutdown_connections 2024-09-16T01:23:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5d9bb90 0x55ffa5d9e000 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d89620 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d890e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f07ebfff640 0 client.26093 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 --2- 172.21.15.78:0/3858286550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d88ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 >> 172.21.15.78:0/3858286550 conn(0x55ffa5d7d060 msgr2=0x55ffa5d82d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 shutdown_connections 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.388+0000 7f0815c09740 1 -- 172.21.15.78:0/3858286550 wait complete. 2024-09-16T01:23:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.389+0000 7f0815c09740 20 client.26093 trim_cache size 0 max 16384 2024-09-16T01:23:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.432+0000 7f0815c09740 1 Processor -- start 2024-09-16T01:23:29.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.437+0000 7f0815c09740 1 -- start start 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.437+0000 7f0815c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.437+0000 7f0815c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.438+0000 7f0815c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5da3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.438+0000 7f0815c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1b3c50 con 0x55ffa5d80af0 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.438+0000 7f0815c09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1b1520 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.438+0000 7f0815c09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1af390 con 0x55ffa5d81410 2024-09-16T01:23:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.439+0000 7f080bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.440+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.440+0000 7f0810c13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52764/0 (socket says 172.21.15.78:52764) 2024-09-16T01:23:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.440+0000 7f0810c13640 1 -- 172.21.15.78:0/3280473908 learned_addr learned my addr 172.21.15.78:0/3280473908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 -- 172.21.15.78:0/3280473908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 msgr2=0x55ffa5d8ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 -- 172.21.15.78:0/3280473908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 msgr2=0x55ffa5da3960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5da3960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 -- 172.21.15.78:0/3280473908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffa5d7dcf0 con 0x55ffa5d80af0 2024-09-16T01:23:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.441+0000 7f0810c13640 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0a0140 tx=0x7f07fc0a4980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8739cacfa9a9b66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.445+0000 7f07f27fc640 1 -- 172.21.15.78:0/3280473908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc01d040 con 0x55ffa5d80af0 2024-09-16T01:23:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.445+0000 7f07f27fc640 1 -- 172.21.15.78:0/3280473908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07fc099d20 con 0x55ffa5d80af0 2024-09-16T01:23:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.445+0000 7f07f27fc640 1 -- 172.21.15.78:0/3280473908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc0ba560 con 0x55ffa5d80af0 2024-09-16T01:23:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.447+0000 7f0815c09740 1 -- 172.21.15.78:0/3280473908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 msgr2=0x55ffa5d89450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.447+0000 7f0815c09740 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0a0140 tx=0x7f07fc0a4980 comp rx=0 tx=0).stop 2024-09-16T01:23:29.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.455+0000 7f0815c09740 1 -- 172.21.15.78:0/3280473908 shutdown_connections 2024-09-16T01:23:29.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.455+0000 7f0815c09740 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5da3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.455+0000 7f0815c09740 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d81410 0x55ffa5d8ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.455+0000 7f0815c09740 1 --2- 172.21.15.78:0/3280473908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d89450 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.455+0000 7f0815c09740 1 -- 172.21.15.78:0/3280473908 >> 172.21.15.78:0/3280473908 conn(0x55ffa5d7d060 msgr2=0x55ffa5d8a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.458+0000 7f0815c09740 1 -- 172.21.15.78:0/3280473908 shutdown_connections 2024-09-16T01:23:29.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.458+0000 7f0815c09740 1 -- 172.21.15.78:0/3280473908 wait complete. 2024-09-16T01:23:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.459+0000 7f0815c09740 1 -- start start 2024-09-16T01:23:29.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.459+0000 7f0815c09740 1 Processor -- start 2024-09-16T01:23:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.459+0000 7f0815c09740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.459+0000 7f0815c09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0815c09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0815c09740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1af9e0 con 0x55ffa5d80af0 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f080bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f080bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:35476/0 (socket says 172.21.15.78:35476) 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f080bfff640 1 -- 172.21.15.78:0/4069655989 learned_addr learned my addr 172.21.15.78:0/4069655989 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0811414640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1b0e50 con 0x55ffa5d81410 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1b00b0 con 0x55ffa5d8bcb0 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.460+0000 7f0815c09740 10 client.0 _ll_register_callbacks cb 0x7f080400a980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 -- 172.21.15.78:0/4069655989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 msgr2=0x55ffa5d8fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 -- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 msgr2=0x55ffa5d8f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffa5d90020 con 0x55ffa5d80af0 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f0810c13640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d941d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f07fc09b980 tx=0x7f07fc0058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c46b211b998fdc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.461+0000 7f07ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.462+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc01d070 con 0x55ffa5d80af0 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.462+0000 7f080bfff640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.462+0000 7f0811414640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.462+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f07fc0ba500 con 0x55ffa5d80af0 2024-09-16T01:23:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.462+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07fc099d20 con 0x55ffa5d80af0 2024-09-16T01:23:29.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.463+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07fc09b0d0 con 0x55ffa5d80af0 2024-09-16T01:23:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.465+0000 7f0815c09740 10 client.16800 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.465+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ffa5e06740 con 0x55ffa5d80af0 2024-09-16T01:23:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07fc099d20 con 0x55ffa5d80af0 2024-09-16T01:23:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f07fc082030 con 0x55ffa5d80af0 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 10 client.16800 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 20 client.16800 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5e06c40 0x55ffa5e09060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 4 client.16800 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:23:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffa5cecd70 con 0x55ffa5e06c40 2024-09-16T01:23:29.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.467+0000 7f080bfff640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5e06c40 0x55ffa5e09060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.471+0000 7f080bfff640 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5e06c40 0x55ffa5e09060 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bbfe841395512bd server_cookie=6337692a88d6d3cd in_seq=0 out_seq=0 2024-09-16T01:23:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.471+0000 7f07ebfff640 10 client.16800 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.544+0000 7f07ebfff640 1 -- 172.21.15.78:0/4069655989 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffa5cecd70 con 0x55ffa5e06c40 2024-09-16T01:23:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.544+0000 7f07ebfff640 10 client.16800 handle_command_reply: tid=0 2024-09-16T01:23:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.544+0000 7f0815c09740 1 client.16800 shutdown 2024-09-16T01:23:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 msgr2=0x55ffa5d941d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d941d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f07fc09b980 tx=0x7f07fc0058f0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 shutdown_connections 2024-09-16T01:23:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55ffa5e06c40 0x55ffa5e09060 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ffa5d8bcb0 0x55ffa5d8fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffa5d81410 0x55ffa5d8f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 --2- 172.21.15.78:0/4069655989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ffa5d80af0 0x55ffa5d941d0 secure :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f07fc09b980 tx=0x7f07fc0058f0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 >> 172.21.15.78:0/4069655989 conn(0x55ffa5d7d060 msgr2=0x55ffa5d8a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.546+0000 7f07ebfff640 0 client.16800 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.547+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 shutdown_connections 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.547+0000 7f0815c09740 1 -- 172.21.15.78:0/4069655989 wait complete. 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.547+0000 7f0815c09740 20 client.16800 trim_cache size 0 max 16384 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:23:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 msgr2=0x7f07f40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 0x7f07f40a5920 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f080c10e870 tx=0x7f0800005a20 comp rx=0 tx=0).stop 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 msgr2=0x7f080c19a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00c970 tx=0x7f07fc00ce30 comp rx=0 tx=0).stop 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 shutdown_connections 2024-09-16T01:23:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f07f40a3460 0x7f07f40a5920 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f080c10e450 0x7f080c19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f080c1011c0 0x7f080c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 --2- 172.21.15.78:0/1445684620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f080c1008a0 0x7f080c19a5e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 >> 172.21.15.78:0/1445684620 conn(0x7f080c0fc6b0 msgr2=0x7f080c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 shutdown_connections 2024-09-16T01:23:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:29.548+0000 7f0812e77640 1 -- 172.21.15.78:0/1445684620 wait complete. 2024-09-16T01:23:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 735.0 s, 644.4 tps, lat 156.601 ms stddev 184.753 2024-09-16T01:23:31.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:30 smithi078 ceph-mon[30937]: pgmap v399: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 546 op/s 2024-09-16T01:23:31.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:30 smithi193 ceph-mon[35701]: pgmap v399: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 546 op/s 2024-09-16T01:23:31.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:30 smithi148 ceph-mon[35030]: pgmap v399: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 546 op/s 2024-09-16T01:23:33.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:32 smithi078 ceph-mon[30937]: pgmap v400: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-09-16T01:23:33.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:32 smithi193 ceph-mon[35701]: pgmap v400: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-09-16T01:23:33.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:32 smithi148 ceph-mon[35030]: pgmap v400: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-09-16T01:23:35.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:34 smithi078 ceph-mon[30937]: pgmap v401: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 514 op/s 2024-09-16T01:23:35.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:34 smithi193 ceph-mon[35701]: pgmap v401: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 514 op/s 2024-09-16T01:23:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:34 smithi148 ceph-mon[35030]: pgmap v401: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 514 op/s 2024-09-16T01:23:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 740.0 s, 522.4 tps, lat 186.795 ms stddev 190.205 2024-09-16T01:23:37.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:36 smithi078 ceph-mon[30937]: pgmap v402: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 543 op/s 2024-09-16T01:23:37.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:36 smithi193 ceph-mon[35701]: pgmap v402: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 543 op/s 2024-09-16T01:23:37.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:36 smithi148 ceph-mon[35030]: pgmap v402: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.9 MiB/s wr, 543 op/s 2024-09-16T01:23:39.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:38 smithi078 ceph-mon[30937]: pgmap v403: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.1 MiB/s wr, 573 op/s 2024-09-16T01:23:39.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:38 smithi193 ceph-mon[35701]: pgmap v403: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.1 MiB/s wr, 573 op/s 2024-09-16T01:23:39.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:38 smithi148 ceph-mon[35030]: pgmap v403: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.1 MiB/s wr, 573 op/s 2024-09-16T01:23:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 745.0 s, 766.8 tps, lat 135.007 ms stddev 165.556 2024-09-16T01:23:41.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:40 smithi078 ceph-mon[30937]: pgmap v404: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-09-16T01:23:41.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:40 smithi193 ceph-mon[35701]: pgmap v404: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-09-16T01:23:41.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:40 smithi148 ceph-mon[35030]: pgmap v404: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-09-16T01:23:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.671+0000 7f468feaf640 1 -- 172.21.15.78:0/3296976921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688100490 msgr2=0x7f4688100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.671+0000 7f468feaf640 1 --2- 172.21.15.78:0/3296976921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688100490 0x7f4688100870 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4684009930 tx=0x7f468402a420 comp rx=0 tx=0).stop 2024-09-16T01:23:42.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 -- 172.21.15.78:0/3296976921 shutdown_connections 2024-09-16T01:23:42.673 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 --2- 172.21.15.78:0/3296976921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46881026f0 0x7f468810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 --2- 172.21.15.78:0/3296976921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4688101d70 0x7f46881021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 --2- 172.21.15.78:0/3296976921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688100490 0x7f4688100870 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 -- 172.21.15.78:0/3296976921 >> 172.21.15.78:0/3296976921 conn(0x7f46880770f0 msgr2=0x7f46880fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 -- 172.21.15.78:0/3296976921 shutdown_connections 2024-09-16T01:23:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.672+0000 7f468feaf640 1 -- 172.21.15.78:0/3296976921 wait complete. 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.673+0000 7f468feaf640 1 Processor -- start 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.673+0000 7f468feaf640 1 -- start start 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.673+0000 7f468feaf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 0x7f468819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.673+0000 7f468feaf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468feaf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 0x7f468819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468feaf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f468810fdd0 con 0x7f4688101d70 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468feaf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f468810fc50 con 0x7f4688100490 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468feaf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f468810ff50 con 0x7f46881026f0 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38544/0 (socket says 172.21.15.78:38544) 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 -- 172.21.15.78:0/751909332 learned_addr learned my addr 172.21.15.78:0/751909332 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468dc4b640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 0x7f468819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468e44c640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 0x7f468819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 -- 172.21.15.78:0/751909332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 msgr2=0x7f468819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 0x7f468819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 -- 172.21.15.78:0/751909332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 msgr2=0x7f468819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 0x7f468819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.677 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 -- 172.21.15.78:0/751909332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468819f680 con 0x7f4688101d70 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468e44c640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 0x7f468819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468dc4b640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 0x7f468819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.674+0000 7f468d44a640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9f0 tx=0x7f467800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.675+0000 7f467effd640 1 -- 172.21.15.78:0/751909332 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467800cd80 con 0x7f4688101d70 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.675+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468819f970 con 0x7f4688101d70 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.675+0000 7f467effd640 1 -- 172.21.15.78:0/751909332 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4678005500 con 0x7f4688101d70 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.675+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46881a7200 con 0x7f4688101d70 2024-09-16T01:23:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.675+0000 7f467effd640 1 -- 172.21.15.78:0/751909332 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4678010690 con 0x7f4688101d70 2024-09-16T01:23:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.678+0000 7f467effd640 1 -- 172.21.15.78:0/751909332 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f46780046c0 con 0x7f4688101d70 2024-09-16T01:23:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.679+0000 7f4692c42740 1 Processor -- start 2024-09-16T01:23:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.680+0000 7f467effd640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 0x7f46740a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.680+0000 7f467effd640 1 -- 172.21.15.78:0/751909332 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f46780d0520 con 0x7f4688101d70 2024-09-16T01:23:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.681+0000 7f4692c42740 1 -- start start 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f468e44c640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 0x7f46740a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f7ffe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffec20 0x55e0f8008f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e0f7f5a2f0 con 0x55e0f7ffe300 2024-09-16T01:23:42.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e0f7f02270 con 0x55e0f80094c0 2024-09-16T01:23:42.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.685+0000 7f4692c42740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e0f7f562b0 con 0x55e0f7ffec20 2024-09-16T01:23:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.687+0000 7f468e44c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.687+0000 7f468e44c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59802/0 (socket says 172.21.15.78:59802) 2024-09-16T01:23:42.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.687+0000 7f468e44c640 1 -- 172.21.15.78:0/986685387 learned_addr learned my addr 172.21.15.78:0/986685387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.691+0000 7f468e44c640 1 -- 172.21.15.78:0/986685387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffec20 msgr2=0x55e0f8008f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.691+0000 7f468e44c640 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffec20 0x55e0f8008f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.691+0000 7f468e44c640 1 -- 172.21.15.78:0/986685387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 msgr2=0x55e0f7ffe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.691+0000 7f468e44c640 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f7ffe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.691+0000 7f468e44c640 1 -- 172.21.15.78:0/986685387 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f7f01340 con 0x55e0f80094c0 2024-09-16T01:23:42.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.693+0000 7f468e44c640 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f468000c5d0 tx=0x7f468000ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b70c9a8f5cbee4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.696+0000 7f468e44c640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 0x7f46740a59a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f468000aab0 tx=0x7f4680015000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.698+0000 7f4673fff640 1 -- 172.21.15.78:0/986685387 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680021070 con 0x55e0f80094c0 2024-09-16T01:23:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.698+0000 7f4673fff640 1 -- 172.21.15.78:0/986685387 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4680005030 con 0x55e0f80094c0 2024-09-16T01:23:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.698+0000 7f4673fff640 1 -- 172.21.15.78:0/986685387 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468001d630 con 0x55e0f80094c0 2024-09-16T01:23:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.699+0000 7f4692c42740 1 -- 172.21.15.78:0/986685387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 msgr2=0x55e0f800b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.699+0000 7f4692c42740 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f468000c5d0 tx=0x7f468000ca90 comp rx=0 tx=0).stop 2024-09-16T01:23:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.704+0000 7f4692c42740 1 -- 172.21.15.78:0/986685387 shutdown_connections 2024-09-16T01:23:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.704+0000 7f4692c42740 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f800b8b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.704+0000 7f4692c42740 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffec20 0x55e0f8008f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.704+0000 7f4692c42740 1 --2- 172.21.15.78:0/986685387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f7ffe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.704+0000 7f4692c42740 1 -- 172.21.15.78:0/986685387 >> 172.21.15.78:0/986685387 conn(0x55e0f7ffa870 msgr2=0x55e0f7ffac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.713+0000 7f4692c42740 1 -- 172.21.15.78:0/986685387 shutdown_connections 2024-09-16T01:23:42.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.717+0000 7f4692c42740 1 -- 172.21.15.78:0/986685387 wait complete. 2024-09-16T01:23:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.718+0000 7f4692c42740 1 -- start start 2024-09-16T01:23:42.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.718+0000 7f4692c42740 1 Processor -- start 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffe300 0x55e0f80064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f8006f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e0f801d0a0 con 0x55e0f7ffec20 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e0f7ee3290 con 0x55e0f80094c0 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e0f801d220 con 0x55e0f7ffe300 2024-09-16T01:23:42.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.722+0000 7f4692c42740 10 client.0 _ll_register_callbacks cb 0x55e0f7f14140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.725+0000 7f468d44a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.725+0000 7f468d44a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38562/0 (socket says 172.21.15.78:38562) 2024-09-16T01:23:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.725+0000 7f468d44a640 1 -- 172.21.15.78:0/68809521 learned_addr learned my addr 172.21.15.78:0/68809521 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.729+0000 7f468e44c640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f8006f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.730+0000 7f468d44a640 1 -- 172.21.15.78:0/68809521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffe300 msgr2=0x55e0f80064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.730+0000 7f468d44a640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffe300 0x55e0f80064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.730+0000 7f468d44a640 1 -- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 msgr2=0x55e0f8006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.730+0000 7f468d44a640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f8006f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.730+0000 7f468d44a640 1 -- 172.21.15.78:0/68809521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f7c6d210 con 0x55e0f7ffec20 2024-09-16T01:23:42.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.736+0000 7f468d44a640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9c0 tx=0x7f467809bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9916a729d81938b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.737+0000 7f46717fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.741+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46780b5060 con 0x55e0f7ffec20 2024-09-16T01:23:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.741+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4680005030 con 0x55e0f7ffec20 2024-09-16T01:23:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.741+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f46780cdad0 con 0x55e0f7ffec20 2024-09-16T01:23:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.747+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f46780ba050 con 0x55e0f7ffec20 2024-09-16T01:23:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.748+0000 7f4692c42740 10 client.16809 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.748+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e0f7d39620 con 0x55e0f7ffec20 2024-09-16T01:23:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.754+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4680005030 con 0x55e0f7ffec20 2024-09-16T01:23:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.754+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f46780a7030 con 0x55e0f7ffec20 2024-09-16T01:23:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.755+0000 7f4692c42740 10 client.16809 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.755+0000 7f4692c42740 10 client.16809 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.756+0000 7f4692c42740 20 client.16809 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.756+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f802a550 0x55e0f802c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.756+0000 7f4692c42740 4 client.16809 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.756+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0f7f67a20 con 0x55e0f802a550 2024-09-16T01:23:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.758+0000 7f468dc4b640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f802a550 0x55e0f802c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.761+0000 7f468dc4b640 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f802a550 0x55e0f802c940 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b440f24556dc453 server_cookie=88cb8d2a3415a400 in_seq=0 out_seq=0 2024-09-16T01:23:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.761+0000 7f46717fa640 10 client.16809 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.767+0000 7f46717fa640 1 -- 172.21.15.78:0/68809521 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e0f7f67a20 con 0x55e0f802a550 2024-09-16T01:23:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.767+0000 7f46717fa640 10 client.16809 handle_command_reply: tid=0 2024-09-16T01:23:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.768+0000 7f4692c42740 1 client.16809 shutdown 2024-09-16T01:23:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.773+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 msgr2=0x55e0f8006a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.773+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9c0 tx=0x7f467809bfe0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 shutdown_connections 2024-09-16T01:23:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f802a550 0x55e0f802c940 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f80094c0 0x55e0f8006f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffec20 0x55e0f8006a30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 --2- 172.21.15.78:0/68809521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f7ffe300 0x55e0f80064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.779+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 >> 172.21.15.78:0/68809521 conn(0x55e0f7ffa870 msgr2=0x55e0f7ff11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.780+0000 7f46717fa640 0 client.16809 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.783+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 shutdown_connections 2024-09-16T01:23:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.785+0000 7f4692c42740 1 -- 172.21.15.78:0/68809521 wait complete. 2024-09-16T01:23:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.787+0000 7f4692c42740 20 client.16809 trim_cache size 0 max 16384 2024-09-16T01:23:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.817+0000 7f4692c42740 1 Processor -- start 2024-09-16T01:23:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.817+0000 7f4692c42740 1 -- start start 2024-09-16T01:23:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f8006da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f80320f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f8032630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4688074010 con 0x55e0f7ffe300 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f46880718e0 con 0x55e0f7ffec20 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f4692c42740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f468806f7d0 con 0x55e0f80094c0 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f468e44c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f8032630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f468e44c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f8032630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:36008/0 (socket says 172.21.15.78:36008) 2024-09-16T01:23:42.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.818+0000 7f468e44c640 1 -- 172.21.15.78:0/3809808355 learned_addr learned my addr 172.21.15.78:0/3809808355 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.820+0000 7f468d44a640 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f80320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.821+0000 7f468dc4b640 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f8006da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.822+0000 7f468dc4b640 1 -- 172.21.15.78:0/3809808355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 msgr2=0x55e0f8032630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.822+0000 7f468dc4b640 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f8032630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.822+0000 7f468dc4b640 1 -- 172.21.15.78:0/3809808355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 msgr2=0x55e0f80320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.822+0000 7f468dc4b640 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f80320f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.822+0000 7f468dc4b640 1 -- 172.21.15.78:0/3809808355 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f8002090 con 0x55e0f7ffe300 2024-09-16T01:23:42.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.823+0000 7f468dc4b640 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f8006da0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4684005940 tx=0x7f4684005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b879ae4b975fcebd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.827+0000 7f46737fe640 1 -- 172.21.15.78:0/3809808355 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684035040 con 0x55e0f7ffe300 2024-09-16T01:23:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.827+0000 7f46737fe640 1 -- 172.21.15.78:0/3809808355 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4684005020 con 0x55e0f7ffe300 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.828+0000 7f4692c42740 1 -- 172.21.15.78:0/3809808355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 msgr2=0x55e0f8006da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.828+0000 7f4692c42740 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f8006da0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4684005940 tx=0x7f4684005d50 comp rx=0 tx=0).stop 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.829+0000 7f4692c42740 1 -- 172.21.15.78:0/3809808355 shutdown_connections 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.829+0000 7f4692c42740 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f8032630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.829+0000 7f4692c42740 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f80320f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.829+0000 7f4692c42740 1 --2- 172.21.15.78:0/3809808355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f8006da0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.829+0000 7f4692c42740 1 -- 172.21.15.78:0/3809808355 >> 172.21.15.78:0/3809808355 conn(0x55e0f7ffa870 msgr2=0x55e0f8008570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.832+0000 7f4692c42740 1 -- 172.21.15.78:0/3809808355 shutdown_connections 2024-09-16T01:23:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.835+0000 7f4692c42740 1 -- 172.21.15.78:0/3809808355 wait complete. 2024-09-16T01:23:42.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.835+0000 7f4692c42740 1 -- start start 2024-09-16T01:23:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.835+0000 7f4692c42740 1 Processor -- start 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f80228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f801dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f801e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f468806fe20 con 0x55e0f7ffe300 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4688071210 con 0x55e0f7ffec20 2024-09-16T01:23:42.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4688070470 con 0x55e0f80094c0 2024-09-16T01:23:42.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f4692c42740 10 client.0 _ll_register_callbacks cb 0x7f4678096f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:42.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f468dc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f80228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f468dc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f80228c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38582/0 (socket says 172.21.15.78:38582) 2024-09-16T01:23:42.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f468dc4b640 1 -- 172.21.15.78:0/1361032654 learned_addr learned my addr 172.21.15.78:0/1361032654 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.839+0000 7f468e44c640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f801e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468d44a640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f801dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468e44c640 1 -- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 msgr2=0x55e0f801dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468e44c640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f801dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468e44c640 1 -- 172.21.15.78:0/1361032654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 msgr2=0x55e0f80228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468e44c640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f80228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.840+0000 7f468e44c640 1 -- 172.21.15.78:0/1361032654 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f801e660 con 0x55e0f80094c0 2024-09-16T01:23:42.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.842+0000 7f468e44c640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f801e120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f468000a380 tx=0x7f468000e7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a37f222b5221cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.843+0000 7f46717fa640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:23:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.843+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680021040 con 0x55e0f80094c0 2024-09-16T01:23:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.843+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f4680005af0 con 0x55e0f80094c0 2024-09-16T01:23:42.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.845+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4684005020 con 0x55e0f80094c0 2024-09-16T01:23:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.850+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4680003970 con 0x55e0f80094c0 2024-09-16T01:23:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.851+0000 7f4692c42740 10 client.26170 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.851+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e0f8094da0 con 0x55e0f80094c0 2024-09-16T01:23:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4684005020 con 0x55e0f80094c0 2024-09-16T01:23:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f468000b040 con 0x55e0f80094c0 2024-09-16T01:23:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 10 client.26170 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:42.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 10 client.26170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 20 client.26170 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f8095480 0x55e0f80978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 4 client.26170 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:23:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.853+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0f7f61d70 con 0x55e0f8095480 2024-09-16T01:23:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.856+0000 7f468dc4b640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f8095480 0x55e0f80978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.857+0000 7f468dc4b640 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f8095480 0x55e0f80978a0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dc160402e2744bb server_cookie=5a48b32eccb63bcb in_seq=0 out_seq=0 2024-09-16T01:23:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.857+0000 7f46717fa640 10 client.26170 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f46717fa640 1 -- 172.21.15.78:0/1361032654 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0f7f61d70 con 0x55e0f8095480 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f46717fa640 10 client.26170 handle_command_reply: tid=0 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 client.26170 shutdown 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 msgr2=0x55e0f801e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f801e120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f468000a380 tx=0x7f468000e7a0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 shutdown_connections 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e0f8095480 0x55e0f80978a0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e0f80094c0 0x55e0f801e120 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e0f7ffec20 0x55e0f801dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 --2- 172.21.15.78:0/1361032654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f7ffe300 0x55e0f80228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 >> 172.21.15.78:0/1361032654 conn(0x55e0f7ffa870 msgr2=0x55e0f7fffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.868+0000 7f46717fa640 0 client.26170 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.869+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 shutdown_connections 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.869+0000 7f4692c42740 1 -- 172.21.15.78:0/1361032654 wait complete. 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.869+0000 7f4692c42740 20 client.26170 trim_cache size 0 max 16384 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:23:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:23:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:23:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 msgr2=0x7f46740a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 0x7f46740a59a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f468000aab0 tx=0x7f4680015000 comp rx=0 tx=0).stop 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 msgr2=0x7f468819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f467800e9f0 tx=0x7f467800edf0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 shutdown_connections 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46740a34e0 0x7f46740a59a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46881026f0 0x7f468819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4688101d70 0x7f468819ac20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 --2- 172.21.15.78:0/751909332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4688100490 0x7f468819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 >> 172.21.15.78:0/751909332 conn(0x7f46880770f0 msgr2=0x7f46881043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 shutdown_connections 2024-09-16T01:23:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:42.870+0000 7f468feaf640 1 -- 172.21.15.78:0/751909332 wait complete. 2024-09-16T01:23:43.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:42 smithi078 ceph-mon[30937]: pgmap v405: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 7.7 MiB/s wr, 657 op/s 2024-09-16T01:23:43.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:42 smithi193 ceph-mon[35701]: pgmap v405: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 7.7 MiB/s wr, 657 op/s 2024-09-16T01:23:43.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:42 smithi148 ceph-mon[35030]: pgmap v405: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 7.7 MiB/s wr, 657 op/s 2024-09-16T01:23:45.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:44 smithi078 ceph-mon[30937]: pgmap v406: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 7.3 MiB/s wr, 620 op/s 2024-09-16T01:23:45.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:44 smithi193 ceph-mon[35701]: pgmap v406: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 7.3 MiB/s wr, 620 op/s 2024-09-16T01:23:45.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:44 smithi148 ceph-mon[35030]: pgmap v406: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 7.3 MiB/s wr, 620 op/s 2024-09-16T01:23:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 750.0 s, 840.4 tps, lat 117.602 ms stddev 132.235 2024-09-16T01:23:47.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:46 smithi078 ceph-mon[30937]: pgmap v407: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 7.5 MiB/s wr, 650 op/s 2024-09-16T01:23:47.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:46 smithi193 ceph-mon[35701]: pgmap v407: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 7.5 MiB/s wr, 650 op/s 2024-09-16T01:23:47.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:46 smithi148 ceph-mon[35030]: pgmap v407: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 7.5 MiB/s wr, 650 op/s 2024-09-16T01:23:47.703 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.4 2024-09-16T01:23:47.704 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:23:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:23:49.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:48 smithi078 ceph-mon[30937]: pgmap v408: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 7.7 MiB/s wr, 660 op/s 2024-09-16T01:23:49.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:48 smithi193 ceph-mon[35701]: pgmap v408: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 7.7 MiB/s wr, 660 op/s 2024-09-16T01:23:49.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:48 smithi148 ceph-mon[35030]: pgmap v408: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 7.7 MiB/s wr, 660 op/s 2024-09-16T01:23:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.203+0000 7fede0975640 1 -- 172.21.15.78:0/397449517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc11b140 msgr2=0x7feddc11d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.203+0000 7fede0975640 1 --2- 172.21.15.78:0/397449517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc11b140 0x7feddc11d530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009860 tx=0x7fedc402a360 comp rx=0 tx=0).stop 2024-09-16T01:23:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.204+0000 7fede0975640 1 -- 172.21.15.78:0/397449517 shutdown_connections 2024-09-16T01:23:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.204+0000 7fede0975640 1 --2- 172.21.15.78:0/397449517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc11b140 0x7feddc11d530 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.205+0000 7fede0975640 1 --2- 172.21.15.78:0/397449517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc109830 0x7feddc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.205+0000 7fede0975640 1 --2- 172.21.15.78:0/397449517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109450 0x7feddc110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.205+0000 7fede0975640 1 -- 172.21.15.78:0/397449517 >> 172.21.15.78:0/397449517 conn(0x7feddc06e140 msgr2=0x7feddc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.210+0000 7fede0975640 1 -- 172.21.15.78:0/397449517 shutdown_connections 2024-09-16T01:23:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.210+0000 7fede0975640 1 -- 172.21.15.78:0/397449517 wait complete. 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 Processor -- start 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 -- start start 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc109450 0x7feddc11a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 0x7feddc1148d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 0x7feddc114e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feddc111330 con 0x7feddc109830 2024-09-16T01:23:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1111b0 con 0x7feddc109450 2024-09-16T01:23:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.211+0000 7fede0975640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1114b0 con 0x7feddc11b140 2024-09-16T01:23:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.214+0000 7fedda59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc109450 0x7feddc11a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.214+0000 7feddad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 0x7feddc114e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.214+0000 7feddad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 0x7feddc114e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:48224/0 (socket says 172.21.15.78:48224) 2024-09-16T01:23:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.214+0000 7feddad9d640 1 -- 172.21.15.78:0/1178104030 learned_addr learned my addr 172.21.15.78:0/1178104030 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.216+0000 7fedd9d9b640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 0x7feddc1148d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.218+0000 7fedd9d9b640 1 -- 172.21.15.78:0/1178104030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 msgr2=0x7feddc114e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.218+0000 7fedd9d9b640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 0x7feddc114e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.218+0000 7fedd9d9b640 1 -- 172.21.15.78:0/1178104030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc109450 msgr2=0x7feddc11a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.218+0000 7fedd9d9b640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc109450 0x7feddc11a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.218+0000 7fedd9d9b640 1 -- 172.21.15.78:0/1178104030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feddc115540 con 0x7feddc109830 2024-09-16T01:23:49.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.220+0000 7fedd9d9b640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 0x7feddc1148d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00eb00 tx=0x7fedcc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.226+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc010430 con 0x7feddc109830 2024-09-16T01:23:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.226+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedcc010ab0 con 0x7feddc109830 2024-09-16T01:23:49.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.226+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc017b70 con 0x7feddc109830 2024-09-16T01:23:49.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.229+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feddc1bc0e0 con 0x7feddc109830 2024-09-16T01:23:49.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.229+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feddc1bc540 con 0x7feddc109830 2024-09-16T01:23:49.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.230+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feddc073b00 con 0x7feddc109830 2024-09-16T01:23:49.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.237+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fedcc017d10 con 0x7feddc109830 2024-09-16T01:23:49.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.238+0000 7fedd37fe640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 0x7fedc80a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:49.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.238+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7fedcc0d0ab0 con 0x7feddc109830 2024-09-16T01:23:49.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.239+0000 7fedda59c640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 0x7fedc80a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:49.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.240+0000 7fedda59c640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 0x7fedc80a5a80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005fd0 tx=0x7fedd4005850 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:49.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.246+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fedcc01d060 con 0x7feddc109830 2024-09-16T01:23:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.662+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feddc109c90 con 0x7feddc109830 2024-09-16T01:23:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.666+0000 7fedd37fe640 1 -- 172.21.15.78:0/1178104030 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fedcc09d0f0 con 0x7feddc109830 2024-09-16T01:23:49.673 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:23:49.673 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:23:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:23:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.671+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 msgr2=0x7fedc80a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.671+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 0x7fedc80a5a80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005fd0 tx=0x7fedd4005850 comp rx=0 tx=0).stop 2024-09-16T01:23:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.671+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 msgr2=0x7feddc1148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.671+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 0x7feddc1148d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00eb00 tx=0x7fedcc00c2d0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 shutdown_connections 2024-09-16T01:23:49.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fedc80a35c0 0x7fedc80a5a80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feddc11b140 0x7feddc114e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feddc109830 0x7feddc1148d0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 --2- 172.21.15.78:0/1178104030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feddc109450 0x7feddc11a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:49.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.674+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 >> 172.21.15.78:0/1178104030 conn(0x7feddc06e140 msgr2=0x7feddc070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.676+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 shutdown_connections 2024-09-16T01:23:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:49.680+0000 7fede0975640 1 -- 172.21.15.78:0/1178104030 wait complete. 2024-09-16T01:23:49.706 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:49 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1178104030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:23:50.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:49 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1178104030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:23:50.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:49 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1178104030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:23:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 755.0 s, 512.6 tps, lat 190.063 ms stddev 249.287 2024-09-16T01:23:51.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:50 smithi078 ceph-mon[30937]: pgmap v409: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 7.2 MiB/s wr, 611 op/s 2024-09-16T01:23:51.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:50 smithi193 ceph-mon[35701]: pgmap v409: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 7.2 MiB/s wr, 611 op/s 2024-09-16T01:23:51.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:50 smithi148 ceph-mon[35030]: pgmap v409: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 7.2 MiB/s wr, 611 op/s 2024-09-16T01:23:52.663 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0' 2024-09-16T01:23:52.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0 2024-09-16T01:23:53.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:52 smithi078 ceph-mon[30937]: pgmap v410: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 7.4 MiB/s wr, 685 op/s 2024-09-16T01:23:53.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:52 smithi193 ceph-mon[35701]: pgmap v410: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 7.4 MiB/s wr, 685 op/s 2024-09-16T01:23:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:23:53.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:52 smithi148 ceph-mon[35030]: pgmap v410: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 7.4 MiB/s wr, 685 op/s 2024-09-16T01:23:53.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.699+0000 7f155c49c640 1 -- 172.21.15.78:0/1068102882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1554109450 msgr2=0x7f1554110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:53.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.699+0000 7f155c49c640 1 --2- 172.21.15.78:0/1068102882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1554109450 0x7f1554110250 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f155001a910 tx=0x7f155003b470 comp rx=0 tx=0).stop 2024-09-16T01:23:53.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.700+0000 7f155c49c640 1 -- 172.21.15.78:0/1068102882 shutdown_connections 2024-09-16T01:23:53.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.700+0000 7f155c49c640 1 --2- 172.21.15.78:0/1068102882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f155411b140 0x7f155411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.700+0000 7f155c49c640 1 --2- 172.21.15.78:0/1068102882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f1554109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.700+0000 7f155c49c640 1 --2- 172.21.15.78:0/1068102882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1554109450 0x7f1554110250 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.700+0000 7f155c49c640 1 -- 172.21.15.78:0/1068102882 >> 172.21.15.78:0/1068102882 conn(0x7f155406e140 msgr2=0x7f155406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:53.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.704+0000 7f155c49c640 1 -- 172.21.15.78:0/1068102882 shutdown_connections 2024-09-16T01:23:53.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.704+0000 7f155c49c640 1 -- 172.21.15.78:0/1068102882 wait complete. 2024-09-16T01:23:53.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.705+0000 7f155c49c640 1 Processor -- start 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.710+0000 7f155c49c640 1 -- start start 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1554109450 0x7f155411a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f155411b140 0x7f1554116e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f15541132f0 con 0x7f155411b140 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1554113170 con 0x7f1554109830 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f155c49c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1554113470 con 0x7f1554109450 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f1559a37640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f1559a37640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47582/0 (socket says 172.21.15.78:47582) 2024-09-16T01:23:53.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.711+0000 7f1559a37640 1 -- 172.21.15.78:0/3822183551 learned_addr learned my addr 172.21.15.78:0/3822183551 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:53.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.714+0000 7f1559a37640 1 -- 172.21.15.78:0/3822183551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1554109450 msgr2=0x7f155411a820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:23:53.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.714+0000 7f1559a37640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1554109450 0x7f155411a820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.714+0000 7f1559a37640 1 -- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f155411b140 msgr2=0x7f1554116e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:53.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.714+0000 7f1559a37640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f155411b140 0x7f1554116e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.714+0000 7f1559a37640 1 -- 172.21.15.78:0/3822183551 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1554117530 con 0x7f1554109830 2024-09-16T01:23:53.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.718+0000 7f1559a37640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f154c00eab0 tx=0x7f154c00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.719+0000 7f154b7fe640 1 -- 172.21.15.78:0/3822183551 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c00ce40 con 0x7f1554109830 2024-09-16T01:23:53.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.719+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15541ab170 con 0x7f1554109830 2024-09-16T01:23:53.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.719+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15541ab570 con 0x7f1554109830 2024-09-16T01:23:53.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.720+0000 7f154b7fe640 1 -- 172.21.15.78:0/3822183551 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c011950 con 0x7f1554109830 2024-09-16T01:23:53.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.720+0000 7f154b7fe640 1 -- 172.21.15.78:0/3822183551 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c01aad0 con 0x7f1554109830 2024-09-16T01:23:53.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.726+0000 7f154b7fe640 1 -- 172.21.15.78:0/3822183551 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f154c011470 con 0x7f1554109830 2024-09-16T01:23:53.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.727+0000 7f154b7fe640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 0x7f15400a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.727+0000 7f154b7fe640 1 -- 172.21.15.78:0/3822183551 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f154c0d0910 con 0x7f1554109830 2024-09-16T01:23:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.732+0000 7f155a238640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 0x7f15400a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.740+0000 7f155a238640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 0x7f15400a5a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f155001a6d0 tx=0x7f1550005fd0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.743+0000 7f155e875740 1 Processor -- start 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.747+0000 7f155e875740 1 -- start start 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.747+0000 7f155e875740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.747+0000 7f155e875740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 0x5585998f2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.747+0000 7f155e875740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155a238640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155a238640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47594/0 (socket says 172.21.15.78:47594) 2024-09-16T01:23:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155a238640 1 -- 172.21.15.78:0/1357349747 learned_addr learned my addr 172.21.15.78:0/1357349747 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55859984d2a0 con 0x5585998e8080 2024-09-16T01:23:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155aa39640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.748+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5585997f4b10 con 0x5585998e7760 2024-09-16T01:23:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.749+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558599849260 con 0x5585998f2940 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.749+0000 7f1559a37640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 0x5585998f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.751+0000 7f1559a37640 1 -- 172.21.15.78:0/1357349747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 msgr2=0x5585998f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.751+0000 7f1559a37640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.751+0000 7f1559a37640 1 -- 172.21.15.78:0/1357349747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 msgr2=0x5585998e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.751+0000 7f1559a37640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.751+0000 7f1559a37640 1 -- 172.21.15.78:0/1357349747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585995beb90 con 0x5585998e8080 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.752+0000 7f1559a37640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 0x5585998f2400 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f154c0cdc40 tx=0x7f154c09c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51c2c1fa4be26b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.753+0000 7f1548ff9640 1 -- 172.21.15.78:0/1357349747 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c09ced0 con 0x5585998e8080 2024-09-16T01:23:53.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.753+0000 7f1548ff9640 1 -- 172.21.15.78:0/1357349747 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c09c7d0 con 0x5585998e8080 2024-09-16T01:23:53.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.753+0000 7f155a238640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:53.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.753+0000 7f155aa39640 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:23:53.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.753+0000 7f1548ff9640 1 -- 172.21.15.78:0/1357349747 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0cdec0 con 0x5585998e8080 2024-09-16T01:23:53.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.754+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 msgr2=0x5585998f2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:53.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.754+0000 7f155e875740 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 0x5585998f2400 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f154c0cdc40 tx=0x7f154c09c1f0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.760+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 shutdown_connections 2024-09-16T01:23:53.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.760+0000 7f155e875740 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.760+0000 7f155e875740 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e8080 0x5585998f2400 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.760+0000 7f155e875740 1 --2- 172.21.15.78:0/1357349747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e7760 0x5585998e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.760+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 >> 172.21.15.78:0/1357349747 conn(0x5585998e3cf0 msgr2=0x5585998e40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:53.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.762+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 shutdown_connections 2024-09-16T01:23:53.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.762+0000 7f155e875740 1 -- 172.21.15.78:0/1357349747 wait complete. 2024-09-16T01:23:53.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.763+0000 7f155e875740 1 -- start start 2024-09-16T01:23:53.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.763+0000 7f155e875740 1 Processor -- start 2024-09-16T01:23:53.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998ef690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998efe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5585998f5560 con 0x5585998e7760 2024-09-16T01:23:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5585997d56e0 con 0x5585998e8080 2024-09-16T01:23:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585998f56e0 con 0x5585998f2940 2024-09-16T01:23:53.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.764+0000 7f155e875740 10 client.0 _ll_register_callbacks cb 0x5585997f9a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:53.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.767+0000 7f155aa39640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.767+0000 7f1559a37640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.767+0000 7f155aa39640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998efe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:48272/0 (socket says 172.21.15.78:48272) 2024-09-16T01:23:53.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.767+0000 7f155aa39640 1 -- 172.21.15.78:0/1291976182 learned_addr learned my addr 172.21.15.78:0/1291976182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:53.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.768+0000 7f155a238640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.775+0000 7f155a238640 1 -- 172.21.15.78:0/1291976182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 msgr2=0x5585998efe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.775+0000 7f155a238640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998efe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.775+0000 7f155a238640 1 -- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 msgr2=0x5585998ef690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.775+0000 7f155a238640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998ef690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.775+0000 7f155a238640 1 -- 172.21.15.78:0/1291976182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585998fea00 con 0x5585998e7760 2024-09-16T01:23:53.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.776+0000 7f155a238640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998f23e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f155001a860 tx=0x7f155003b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c89d854c98d6d17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.777+0000 7f153dffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:53.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.779+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550004240 con 0x5585998e7760 2024-09-16T01:23:53.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.779+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f15500043e0 con 0x5585998e7760 2024-09-16T01:23:53.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.783+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f154c09c7d0 con 0x5585998e7760 2024-09-16T01:23:53.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.787+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1550005040 con 0x5585998e7760 2024-09-16T01:23:53.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.788+0000 7f155e875740 10 client.16830 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:53.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.788+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585999011e0 con 0x5585998e7760 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.791+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155005e070 con 0x5585998e7760 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.792+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f15500035f0 con 0x5585998e7760 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 10 client.16830 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 10 client.16830 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 20 client.16830 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5585999026a0 0x558599904ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 4 client.16830 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.793+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f154c09c7d0 con 0x5585999026a0 2024-09-16T01:23:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.797+0000 7f1559a37640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5585999026a0 0x558599904ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.802+0000 7f1559a37640 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5585999026a0 0x558599904ac0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111efacd4d4bfb1e server_cookie=9c43e4c469b31a22 in_seq=0 out_seq=0 2024-09-16T01:23:53.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.802+0000 7f153dffb640 10 client.16830 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:53.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.802+0000 7f153dffb640 1 -- 172.21.15.78:0/1291976182 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f154c09c7d0 con 0x5585999026a0 2024-09-16T01:23:53.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.802+0000 7f153dffb640 10 client.16830 handle_command_reply: tid=0 2024-09-16T01:23:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.804+0000 7f155e875740 1 client.16830 shutdown 2024-09-16T01:23:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.805+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 msgr2=0x5585998f23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.805+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998f23e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f155001a860 tx=0x7f155003b470 comp rx=0 tx=0).stop 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 shutdown_connections 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5585999026a0 0x558599904ac0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998f2940 0x5585998efe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998ef690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 --2- 172.21.15.78:0/1291976182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998f23e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f153dffb640 0 client.16830 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:53.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.806+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 >> 172.21.15.78:0/1291976182 conn(0x5585998e3cf0 msgr2=0x5585998e9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:53.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.807+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 shutdown_connections 2024-09-16T01:23:53.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.809+0000 7f155e875740 1 -- 172.21.15.78:0/1291976182 wait complete. 2024-09-16T01:23:53.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.810+0000 7f155e875740 20 client.16830 trim_cache size 0 max 16384 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.841+0000 7f155e875740 1 Processor -- start 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.841+0000 7f155e875740 1 -- start start 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.841+0000 7f155e875740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.841+0000 7f155e875740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e8080 0x55859990a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.841+0000 7f155e875740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998f2940 0x55859990a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155e875740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f15380127f0 con 0x5585998e7760 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155e875740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f15380100c0 con 0x5585998f2940 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f1559a37640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e8080 0x55859990a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155e875740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f153800dfc0 con 0x5585998e8080 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35914/0 (socket says 172.21.15.78:35914) 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 -- 172.21.15.78:0/2621921797 learned_addr learned my addr 172.21.15.78:0/2621921797 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 -- 172.21.15.78:0/2621921797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e8080 msgr2=0x55859990a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e8080 0x55859990a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 -- 172.21.15.78:0/2621921797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998f2940 msgr2=0x55859990a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998f2940 0x55859990a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.842+0000 7f155a238640 1 -- 172.21.15.78:0/2621921797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585998ef630 con 0x5585998e7760 2024-09-16T01:23:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.843+0000 7f155a238640 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f1550003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6804e880146667f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.843+0000 7f153ffff640 1 -- 172.21.15.78:0/2621921797 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550071070 con 0x5585998e7760 2024-09-16T01:23:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.844+0000 7f153ffff640 1 -- 172.21.15.78:0/2621921797 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155005e070 con 0x5585998e7760 2024-09-16T01:23:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.844+0000 7f153ffff640 1 -- 172.21.15.78:0/2621921797 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15500044e0 con 0x5585998e7760 2024-09-16T01:23:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 -- 172.21.15.78:0/2621921797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 msgr2=0x5585998ee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f1550003e30 comp rx=0 tx=0).stop 2024-09-16T01:23:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 -- 172.21.15.78:0/2621921797 shutdown_connections 2024-09-16T01:23:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998f2940 0x55859990a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e8080 0x55859990a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 --2- 172.21.15.78:0/2621921797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998e7760 0x5585998ee670 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.845+0000 7f155e875740 1 -- 172.21.15.78:0/2621921797 >> 172.21.15.78:0/2621921797 conn(0x5585998e3cf0 msgr2=0x5585998edf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.846+0000 7f155e875740 1 -- 172.21.15.78:0/2621921797 shutdown_connections 2024-09-16T01:23:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.846+0000 7f155e875740 1 -- 172.21.15.78:0/2621921797 wait complete. 2024-09-16T01:23:53.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.846+0000 7f155e875740 1 -- start start 2024-09-16T01:23:53.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 Processor -- start 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 0x5585998f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998f2940 0x5585998f4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f153800e580 con 0x5585998f2940 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f153800f9f0 con 0x5585998e8080 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f153800ec50 con 0x5585998e7760 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.847+0000 7f155e875740 10 client.0 _ll_register_callbacks cb 0x7f1550043a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.848+0000 7f1559a37640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.848+0000 7f1559a37640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:47624/0 (socket says 172.21.15.78:47624) 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.848+0000 7f1559a37640 1 -- 172.21.15.78:0/1366470225 learned_addr learned my addr 172.21.15.78:0/1366470225 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.848+0000 7f155a238640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 0x5585998f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 -- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 msgr2=0x5585998f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 -- 172.21.15.78:0/1366470225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998f2940 msgr2=0x5585998f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998f2940 0x5585998f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 -- 172.21.15.78:0/1366470225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585998f8f40 con 0x5585998e7760 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f155a238640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 0x5585998f3aa0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1550003ae0 tx=0x7f1550004480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb653514756da2d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.849+0000 7f153dffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.850+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550071040 con 0x5585998e7760 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.850+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f15500051e0 con 0x5585998e7760 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.850+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155005e070 con 0x5585998e7760 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.851+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f15500053e0 con 0x5585998e7760 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.852+0000 7f155e875740 10 client.26182 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.852+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55859996d670 con 0x5585998e7760 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.852+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155005e070 con 0x5585998e7760 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f155006b070 con 0x5585998e7760 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 10 client.26182 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 10 client.26182 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 20 client.26182 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55859996db60 0x55859996ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 4 client.26182 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}] 2024-09-16T01:23:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}) -- 0x55859985a9d0 con 0x55859996db60 2024-09-16T01:23:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.853+0000 7f1559a37640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55859996db60 0x55859996ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.854+0000 7f153dffb640 10 client.26182 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:53.854+0000 7f1559a37640 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55859996db60 0x55859996ff50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e5b0e80febfedbb server_cookie=544e753184a791e2 in_seq=0 out_seq=0 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f153dffb640 1 -- 172.21.15.78:0/1366470225 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55859985a9d0 con 0x55859996db60 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f153dffb640 10 client.26182 handle_command_reply: tid=0 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 client.26182 shutdown 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 msgr2=0x5585998f3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 0x5585998f3aa0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1550003ae0 tx=0x7f1550004480 comp rx=0 tx=0).stop 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 shutdown_connections 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55859996db60 0x55859996ff50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585998f2940 0x5585998f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f153dffb640 0 client.26182 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585998e8080 0x5585998f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 --2- 172.21.15.78:0/1366470225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585998e7760 0x5585998f3aa0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 >> 172.21.15.78:0/1366470225 conn(0x5585998e3cf0 msgr2=0x5585998ee780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 shutdown_connections 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 1 -- 172.21.15.78:0/1366470225 wait complete. 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.294+0000 7f155e875740 20 client.26182 trim_cache size 0 max 16384 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 46, 2024-09-16T01:23:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "3ee6b2b6": { 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 46, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 131.4, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:23:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:23:54.300 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 msgr2=0x7f15400a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 0x7f15400a5a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f155001a6d0 tx=0x7f1550005fd0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 msgr2=0x7f15541168c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f154c00eab0 tx=0x7f154c00ef70 comp rx=0 tx=0).stop 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 shutdown_connections 2024-09-16T01:23:54.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f15400a35c0 0x7f15400a5a80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f155411b140 0x7f1554116e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1554109830 0x7f15541168c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 --2- 172.21.15.78:0/3822183551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1554109450 0x7f155411a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 >> 172.21.15.78:0/3822183551 conn(0x7f155406e140 msgr2=0x7f1554072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 shutdown_connections 2024-09-16T01:23:54.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:54.295+0000 7f155c49c640 1 -- 172.21.15.78:0/3822183551 wait complete. 2024-09-16T01:23:55.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:54 smithi078 ceph-mon[30937]: pgmap v411: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.8 MiB/s wr, 647 op/s 2024-09-16T01:23:55.038 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3ee6b2b6', quiesce took 7.3 seconds. Will release after: 4.1000000000000005 2024-09-16T01:23:55.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:54 smithi193 ceph-mon[35701]: pgmap v411: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.8 MiB/s wr, 647 op/s 2024-09-16T01:23:55.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:54 smithi148 ceph-mon[35030]: pgmap v411: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 6.8 MiB/s wr, 647 op/s 2024-09-16T01:23:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 760.0 s, 323.1 tps, lat 185.681 ms stddev 237.094 2024-09-16T01:23:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 -- 172.21.15.78:0/2499477961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1810c0d0 msgr2=0x7f1c1810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 --2- 172.21.15.78:0/2499477961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1810c0d0 0x7f1c1810e590 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009900 tx=0x7f1c1002a480 comp rx=0 tx=0).stop 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 -- 172.21.15.78:0/2499477961 shutdown_connections 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 --2- 172.21.15.78:0/2499477961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c1810c0d0 0x7f1c1810e590 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 --2- 172.21.15.78:0/2499477961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c181035d0 0x7f1c1810bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 --2- 172.21.15.78:0/2499477961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 0x7f1c18103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 -- 172.21.15.78:0/2499477961 >> 172.21.15.78:0/2499477961 conn(0x7f1c180fc6c0 msgr2=0x7f1c180feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 -- 172.21.15.78:0/2499477961 shutdown_connections 2024-09-16T01:23:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.965+0000 7f1c2074b640 1 -- 172.21.15.78:0/2499477961 wait complete. 2024-09-16T01:23:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 Processor -- start 2024-09-16T01:23:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 -- start start 2024-09-16T01:23:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 0x7f1c18198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 0x7f1c1819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c18111ef0 con 0x7f1c181035d0 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c18111d70 con 0x7f1c18102cb0 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c2074b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c18112070 con 0x7f1c1810c0d0 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.966+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40266/0 (socket says 172.21.15.78:40266) 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 -- 172.21.15.78:0/992814308 learned_addr learned my addr 172.21.15.78:0/992814308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 0x7f1c18198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 0x7f1c1819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 -- 172.21.15.78:0/992814308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 msgr2=0x7f1c1819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 0x7f1c1819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 -- 172.21.15.78:0/992814308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 msgr2=0x7f1c18198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 0x7f1c18198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 -- 172.21.15.78:0/992814308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1819d2f0 con 0x7f1c181035d0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 0x7f1c1819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c0f7fe640 1 -- 172.21.15.78:0/992814308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800cd80 con 0x7f1c181035d0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c0f7fe640 1 -- 172.21.15.78:0/992814308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c08005500 con 0x7f1c181035d0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1819d5e0 con 0x7f1c181035d0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c0f7fe640 1 -- 172.21.15.78:0/992814308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08010690 con 0x7f1c181035d0 2024-09-16T01:23:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.967+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c181a4e70 con 0x7f1c181035d0 2024-09-16T01:23:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.969+0000 7f1c0f7fe640 1 -- 172.21.15.78:0/992814308 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f1c080046c0 con 0x7f1c181035d0 2024-09-16T01:23:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 Processor -- start 2024-09-16T01:23:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 -- start start 2024-09-16T01:23:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 0x55a68b58ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 0x55a68b597770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a68b4f12f0 con 0x55a68b597cb0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a68b499270 con 0x55a68b58d410 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c23122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a68b4ed2b0 con 0x55a68b58caf0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40278/0 (socket says 172.21.15.78:40278) 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c1dce6640 1 -- 172.21.15.78:0/1821541268 learned_addr learned my addr 172.21.15.78:0/1821541268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c0f7fe640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 0x7f1c040a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 0x55a68b597770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.970+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 0x55a68b58ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 0x7f1c040a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c0f7fe640 1 -- 172.21.15.78:0/992814308 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f1c080d04a0 con 0x7f1c181035d0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 -- 172.21.15.78:0/1821541268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 msgr2=0x55a68b58ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 0x55a68b58ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 -- 172.21.15.78:0/1821541268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 msgr2=0x55a68b597770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 0x55a68b597770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 -- 172.21.15.78:0/1821541268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a68b498340 con 0x55a68b597cb0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 0x55a68b597770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 0x55a68b58ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1c0809c160 tx=0x7f1c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f381780fa3e06ee4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 0x7f1c040a5920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1c100057c0 tx=0x7f1c10009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c0cff9640 1 -- 172.21.15.78:0/1821541268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0809bfa0 con 0x55a68b597cb0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c0cff9640 1 -- 172.21.15.78:0/1821541268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0809cb60 con 0x55a68b597cb0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.971+0000 7f1c0cff9640 1 -- 172.21.15.78:0/1821541268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0809bdc0 con 0x55a68b597cb0 2024-09-16T01:23:55.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- 172.21.15.78:0/1821541268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 msgr2=0x55a68b59a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1c0809c160 tx=0x7f1c08004a40 comp rx=0 tx=0).stop 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- 172.21.15.78:0/1821541268 shutdown_connections 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b59a0a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58d410 0x55a68b597770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 --2- 172.21.15.78:0/1821541268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58caf0 0x55a68b58ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- 172.21.15.78:0/1821541268 >> 172.21.15.78:0/1821541268 conn(0x55a68b589060 msgr2=0x55a68b589440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- 172.21.15.78:0/1821541268 shutdown_connections 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- 172.21.15.78:0/1821541268 wait complete. 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 -- start start 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.972+0000 7f1c23122740 1 Processor -- start 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b594ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58d410 0x55a68b5950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a68b59a880 con 0x55a68b597cb0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a68b47a290 con 0x55a68b58caf0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a68b59aa00 con 0x55a68b58d410 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 10 client.0 _ll_register_callbacks cb 0x55a68b4ab140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40288/0 (socket says 172.21.15.78:40288) 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 -- 172.21.15.78:0/2453792440 learned_addr learned my addr 172.21.15.78:0/2453792440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58d410 0x55a68b5950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b594ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 -- 172.21.15.78:0/2453792440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58d410 msgr2=0x55a68b5950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58d410 0x55a68b5950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 -- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 msgr2=0x55a68b594ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b594ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 -- 172.21.15.78:0/2453792440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a68b204210 con 0x55a68b597cb0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b594ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1c10033000 tx=0x7f1c1002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bb15e02246f1e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1004b020 con 0x55a68b597cb0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c0809cb60 con 0x55a68b597cb0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.973+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f1c10003d70 con 0x55a68b597cb0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1c10056050 con 0x55a68b597cb0 2024-09-16T01:23:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 10 client.16845 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a68b2d0620 con 0x55a68b597cb0 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0809cb60 con 0x55a68b597cb0 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1c1003a040 con 0x55a68b597cb0 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 10 client.16845 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 20 client.16845 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:55.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b5a7c00 0x55a68b5aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 4 client.16845 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a68b4fea20 con 0x55a68b5a7c00 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.974+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b5a7c00 0x55a68b5aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.975+0000 7f1c01ffb640 10 client.16845 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.975+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b5a7c00 0x55a68b5aa050 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da05a27ce7ca5f42 server_cookie=84c4448e4fd2cec6 in_seq=0 out_seq=0 2024-09-16T01:23:55.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.975+0000 7f1c01ffb640 1 -- 172.21.15.78:0/2453792440 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a68b4fea20 con 0x55a68b5a7c00 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.975+0000 7f1c01ffb640 10 client.16845 handle_command_reply: tid=0 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 client.16845 shutdown 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 msgr2=0x55a68b595620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1c10033000 tx=0x7f1c1002a480 comp rx=0 tx=0).stop 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 shutdown_connections 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b5a7c00 0x55a68b5aa050 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b597cb0 0x55a68b595620 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b58d410 0x55a68b5950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 --2- 172.21.15.78:0/2453792440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b594ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c01ffb640 0 client.16845 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 >> 172.21.15.78:0/2453792440 conn(0x55a68b589060 msgr2=0x55a68b58ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 shutdown_connections 2024-09-16T01:23:55.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 1 -- 172.21.15.78:0/2453792440 wait complete. 2024-09-16T01:23:55.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.976+0000 7f1c23122740 20 client.16845 trim_cache size 0 max 16384 2024-09-16T01:23:55.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 Processor -- start 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 -- start start 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b595450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b5af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181b18a0 con 0x55a68b58d410 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181af170 con 0x55a68b58caf0 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c23122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181acfe0 con 0x55a68b597cb0 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.990+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40290/0 (socket says 172.21.15.78:40290) 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3682346261 learned_addr learned my addr 172.21.15.78:0/3682346261 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b595450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1ece8640 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b5af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3682346261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 msgr2=0x55a68b5af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b5af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3682346261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 msgr2=0x55a68b595450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b595450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3682346261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a68b5aa6d0 con 0x55a68b58d410 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9c0 tx=0x7f1c080a5150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f46e6dff2a4108b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c03fff640 1 -- 172.21.15.78:0/3682346261 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080a0810 con 0x55a68b58d410 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c03fff640 1 -- 172.21.15.78:0/3682346261 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0809cb60 con 0x55a68b58d410 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.991+0000 7f1c03fff640 1 -- 172.21.15.78:0/3682346261 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c080d0160 con 0x55a68b58d410 2024-09-16T01:23:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- 172.21.15.78:0/3682346261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 msgr2=0x55a68b596b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9c0 tx=0x7f1c080a5150 comp rx=0 tx=0).stop 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- 172.21.15.78:0/3682346261 shutdown_connections 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b5af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b596b10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- 172.21.15.78:0/3682346261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b595450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- 172.21.15.78:0/3682346261 >> 172.21.15.78:0/3682346261 conn(0x55a68b589060 msgr2=0x55a68b596400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- 172.21.15.78:0/3682346261 shutdown_connections 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- 172.21.15.78:0/3682346261 wait complete. 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- start start 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 Processor -- start 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b59b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b610970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181ad630 con 0x55a68b58d410 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181aeaa0 con 0x55a68b58caf0 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181add00 con 0x55a68b597cb0 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c23122740 10 client.0 _ll_register_callbacks cb 0x7f1c08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c1dce6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40306/0 (socket says 172.21.15.78:40306) 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3527244229 learned_addr learned my addr 172.21.15.78:0/3527244229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.992+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b59b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3527244229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 msgr2=0x55a68b610970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b610970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 msgr2=0x55a68b59b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:23:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b59b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 -- 172.21.15.78:0/3527244229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a68b589cf0 con 0x55a68b58d410 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b59b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c1dce6640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1c080d3c60 tx=0x7f1c080d0850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d437e5f3f38289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0809c460 con 0x55a68b58d410 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f1c080a4c40 con 0x55a68b58d410 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c0809cb60 con 0x55a68b58d410 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1c080a4ea0 con 0x55a68b58d410 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 10 client.16851 fetch_fsmap learned FSMap version 31 2024-09-16T01:23:56.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a68b6120d0 con 0x55a68b58d410 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0809cb60 con 0x55a68b58d410 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1c0809b8b0 con 0x55a68b58d410 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 10 client.16851 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 20 client.16851 populate_metadata read hostname 'smithi078' 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b612300 0x55a68b6146f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 4 client.16851 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.993+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a68b4f8d70 con 0x55a68b612300 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.994+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b612300 0x55a68b6146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.994+0000 7f1c01ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:56.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:55.994+0000 7f1c1e4e7640 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b612300 0x55a68b6146f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57929203d117a195 server_cookie=e0c4189f514c33a6 in_seq=0 out_seq=0 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c01ffb640 1 -- 172.21.15.78:0/3527244229 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a68b4f8d70 con 0x55a68b612300 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c01ffb640 10 client.16851 handle_command_reply: tid=0 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 client.16851 shutdown 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 msgr2=0x55a68b59b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1c080d3c60 tx=0x7f1c080d0850 comp rx=0 tx=0).stop 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 shutdown_connections 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a68b612300 0x55a68b6146f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a68b597cb0 0x55a68b610970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c01ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:23:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a68b58d410 0x55a68b59b730 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 --2- 172.21.15.78:0/3527244229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a68b58caf0 0x55a68b59b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 >> 172.21.15.78:0/3527244229 conn(0x55a68b589060 msgr2=0x55a68b5881c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 shutdown_connections 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 1 -- 172.21.15.78:0/3527244229 wait complete. 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.002+0000 7f1c23122740 20 client.16851 trim_cache size 0 max 16384 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:23:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 msgr2=0x7f1c040a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 0x7f1c040a5920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1c100057c0 tx=0x7f1c10009260 comp rx=0 tx=0).stop 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 msgr2=0x7f1c18198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800edf0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 shutdown_connections 2024-09-16T01:23:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c040a3460 0x7f1c040a5920 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c1810c0d0 0x7f1c1819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c181035d0 0x7f1c18198840 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 --2- 172.21.15.78:0/992814308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c18102cb0 0x7f1c18198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 >> 172.21.15.78:0/992814308 conn(0x7f1c180fc6c0 msgr2=0x7f1c180fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 shutdown_connections 2024-09-16T01:23:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:23:56.003+0000 7f1c2074b640 1 -- 172.21.15.78:0/992814308 wait complete. 2024-09-16T01:23:57.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:56 smithi078 ceph-mon[30937]: pgmap v412: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-09-16T01:23:57.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:56 smithi193 ceph-mon[35701]: pgmap v412: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-09-16T01:23:57.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:56 smithi148 ceph-mon[35030]: pgmap v412: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-09-16T01:23:59.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:23:58 smithi078 ceph-mon[30937]: pgmap v413: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 6.8 MiB/s wr, 645 op/s 2024-09-16T01:23:59.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:23:58 smithi193 ceph-mon[35701]: pgmap v413: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 6.8 MiB/s wr, 645 op/s 2024-09-16T01:23:59.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:23:58 smithi148 ceph-mon[35030]: pgmap v413: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 6.8 MiB/s wr, 645 op/s 2024-09-16T01:23:59.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:23:59.398 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:23:59.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 -- 172.21.15.78:0/2076508831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118102d90 msgr2=0x7fd118103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:59.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 --2- 172.21.15.78:0/2076508831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118102d90 0x7fd118103170 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd108009930 tx=0x7fd10802a420 comp rx=0 tx=0).stop 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 -- 172.21.15.78:0/2076508831 shutdown_connections 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 --2- 172.21.15.78:0/2076508831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 0x7fd11810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 --2- 172.21.15.78:0/2076508831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118103740 0x7fd118107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 --2- 172.21.15.78:0/2076508831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118102d90 0x7fd118103170 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 -- 172.21.15.78:0/2076508831 >> 172.21.15.78:0/2076508831 conn(0x7fd1180fc7d0 msgr2=0x7fd1180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 -- 172.21.15.78:0/2076508831 shutdown_connections 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.899+0000 7fd11dac4640 1 -- 172.21.15.78:0/2076508831 wait complete. 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 Processor -- start 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 -- start start 2024-09-16T01:23:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118102d90 0x7fd1181985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 0x7fd11819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd118112090 con 0x7fd118103740 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd118111f10 con 0x7fd118102d90 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd11dac4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd118112210 con 0x7fd118108290 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd10effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd10effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40328/0 (socket says 172.21.15.78:40328) 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.900+0000 7fd10effd640 1 -- 172.21.15.78:0/46752516 learned_addr learned my addr 172.21.15.78:0/46752516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd1177fe640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118102d90 0x7fd1181985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:59.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd117fff640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 0x7fd11819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 -- 172.21.15.78:0/46752516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 msgr2=0x7fd11819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 0x7fd11819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 -- 172.21.15.78:0/46752516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118102d90 msgr2=0x7fd1181985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118102d90 0x7fd1181985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 -- 172.21.15.78:0/46752516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11819d5a0 con 0x7fd118103740 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd10effd640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd10000d770 tx=0x7fd10000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd100017070 con 0x7fd118103740 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11819d890 con 0x7fd118103740 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd100009c30 con 0x7fd118103740 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10000f620 con 0x7fd118103740 2024-09-16T01:23:59.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.901+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1181a5170 con 0x7fd118103740 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.902+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd11806a7f0 con 0x7fd118103740 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.903+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fd10001a070 con 0x7fd118103740 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.904+0000 7fd1157fa640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 0x7fd1040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.904+0000 7fd1177fe640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 0x7fd1040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.904+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7fd1000cfc90 con 0x7fd118103740 2024-09-16T01:23:59.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.905+0000 7fd1177fe640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 0x7fd1040a5870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd108000c00 tx=0x7fd108005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:23:59.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:23:59.907+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd100009de0 con 0x7fd118103740 2024-09-16T01:24:00.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.106+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd11810a2a0 con 0x7fd118103740 2024-09-16T01:24:00.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.107+0000 7fd1157fa640 1 -- 172.21.15.78:0/46752516 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fd10009c2b0 con 0x7fd118103740 2024-09-16T01:24:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:24:00.110 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:24:00.110 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:24:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 msgr2=0x7fd1040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 0x7fd1040a5870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd108000c00 tx=0x7fd108005a20 comp rx=0 tx=0).stop 2024-09-16T01:24:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 msgr2=0x7fd118198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:00.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd10000d770 tx=0x7fd10000dc30 comp rx=0 tx=0).stop 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 shutdown_connections 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fd1040a33b0 0x7fd1040a5870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd118108290 0x7fd11819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd118103740 0x7fd118198b30 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 --2- 172.21.15.78:0/46752516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd118102d90 0x7fd1181985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 >> 172.21.15.78:0/46752516 conn(0x7fd1180fc7d0 msgr2=0x7fd118106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 shutdown_connections 2024-09-16T01:24:00.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:00.109+0000 7fd11dac4640 1 -- 172.21.15.78:0/46752516 wait complete. 2024-09-16T01:24:00.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:00 smithi078 ceph-mon[30937]: pgmap v414: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 5.2 MiB/s wr, 541 op/s 2024-09-16T01:24:00.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:00 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/46752516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:00.817 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 3ee6b2b6 --release --await' 2024-09-16T01:24:00.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 3ee6b2b6 --release --await 2024-09-16T01:24:00.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 765.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:24:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:00 smithi193 ceph-mon[35701]: pgmap v414: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 5.2 MiB/s wr, 541 op/s 2024-09-16T01:24:01.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:00 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/46752516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:01.080 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:24:01.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:00 smithi148 ceph-mon[35030]: pgmap v414: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 5.2 MiB/s wr, 541 op/s 2024-09-16T01:24:01.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:00 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/46752516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 -- 172.21.15.78:0/2683029229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894101420 msgr2=0x7f689410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 --2- 172.21.15.78:0/2683029229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894101420 0x7f689410d950 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f688c009cd0 tx=0x7f688c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 -- 172.21.15.78:0/2683029229 shutdown_connections 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 --2- 172.21.15.78:0/2683029229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894101420 0x7f689410d950 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 --2- 172.21.15.78:0/2683029229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894100a80 0x7f6894100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 --2- 172.21.15.78:0/2683029229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f6894106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 -- 172.21.15.78:0/2683029229 >> 172.21.15.78:0/2683029229 conn(0x7f68940fc7a0 msgr2=0x7f68940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 -- 172.21.15.78:0/2683029229 shutdown_connections 2024-09-16T01:24:01.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.566+0000 7f689be8d640 1 -- 172.21.15.78:0/2683029229 wait complete. 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 Processor -- start 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 -- start start 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 0x7f68941988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f689419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6894110e80 con 0x7f6894100a80 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6894110d00 con 0x7f6894106a80 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689be8d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6894111000 con 0x7f6894101420 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40346/0 (socket says 172.21.15.78:40346) 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 -- 172.21.15.78:0/2179299272 learned_addr learned my addr 172.21.15.78:0/2179299272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689a42a640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f689419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899428640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 0x7f68941988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 -- 172.21.15.78:0/2179299272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 msgr2=0x7f68941988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 0x7f68941988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 -- 172.21.15.78:0/2179299272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 msgr2=0x7f689419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f689419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899c29640 1 -- 172.21.15.78:0/2179299272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689419d320 con 0x7f6894100a80 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f689a42a640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f689419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.567+0000 7f6899428640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 0x7f68941988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f6899c29640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f689000d950 tx=0x7f689000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f688affd640 1 -- 172.21.15.78:0/2179299272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890017070 con 0x7f6894100a80 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f688affd640 1 -- 172.21.15.78:0/2179299272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6890009c30 con 0x7f6894100a80 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f688affd640 1 -- 172.21.15.78:0/2179299272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689000f6b0 con 0x7f6894100a80 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689419d610 con 0x7f6894100a80 2024-09-16T01:24:01.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.568+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68941a4ef0 con 0x7f6894100a80 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f688affd640 1 -- 172.21.15.78:0/2179299272 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f68900109d0 con 0x7f6894100a80 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 Processor -- start 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 -- start start 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0ef1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0ef4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0e4c620 con 0x55c0e0ee7060 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0df3e50 con 0x55c0e0ee7980 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689e266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0e485e0 con 0x55c0e0ef2240 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689a42a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689a42a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40362/0 (socket says 172.21.15.78:40362) 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.570+0000 7f689a42a640 1 -- 172.21.15.78:0/2442509054 learned_addr learned my addr 172.21.15.78:0/2442509054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:01.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f6899c29640 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0ef1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f6899428640 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0ef4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 -- 172.21.15.78:0/2442509054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 msgr2=0x55c0e0ef4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0ef4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 -- 172.21.15.78:0/2442509054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 msgr2=0x55c0e0ef1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0ef1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 -- 172.21.15.78:0/2442509054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0e0de4d40 con 0x55c0e0ee7060 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f688affd640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 0x7f68800a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f689a42a640 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f688c0315f0 tx=0x7f688c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6656d5d38486badb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f6899c29640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 0x7f68800a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f687ffff640 1 -- 172.21.15.78:0/2442509054 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c031ac0 con 0x55c0e0ee7060 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f688affd640 1 -- 172.21.15.78:0/2179299272 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f68900cfd40 con 0x7f6894100a80 2024-09-16T01:24:01.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f687ffff640 1 -- 172.21.15.78:0/2442509054 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c036040 con 0x55c0e0ee7060 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.571+0000 7f687ffff640 1 -- 172.21.15.78:0/2442509054 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c0305a0 con 0x55c0e0ee7060 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- 172.21.15.78:0/2442509054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 msgr2=0x55c0e0ee7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f6899c29640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 0x7f68800a58f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55c0e0ee4450 tx=0x7f689000bf60 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f688c0315f0 tx=0x7f688c009a20 comp rx=0 tx=0).stop 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- 172.21.15.78:0/2442509054 shutdown_connections 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0ef4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0ef1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 --2- 172.21.15.78:0/2442509054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee7440 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- 172.21.15.78:0/2442509054 >> 172.21.15.78:0/2442509054 conn(0x55c0e0ee35f0 msgr2=0x55c0e0ee39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- 172.21.15.78:0/2442509054 shutdown_connections 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- 172.21.15.78:0/2442509054 wait complete. 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.572+0000 7f689e266740 1 -- start start 2024-09-16T01:24:01.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 Processor -- start 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0eef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eefbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0ef4e10 con 0x55c0e0ee7060 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0dd4f10 con 0x55c0e0ee7980 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c0e0ef4f90 con 0x55c0e0ef2240 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689e266740 10 client.0 _ll_register_callbacks cb 0x55c0e0df8db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40374/0 (socket says 172.21.15.78:40374) 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 -- 172.21.15.78:0/2562702286 learned_addr learned my addr 172.21.15.78:0/2562702286 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eefbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f6899428640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0eef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 -- 172.21.15.78:0/2562702286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 msgr2=0x55c0e0eefbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eefbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 -- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 msgr2=0x55c0e0eef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0eef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 -- 172.21.15.78:0/2562702286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0e0b5b580 con 0x55c0e0ee7060 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f6899428640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0eef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f689a42a640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f688c004480 tx=0x7f688c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275bc4f3d650348e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.573+0000 7f687d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c003b80 con 0x55c0e0ee7060 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f688c003d20 con 0x55c0e0ee7060 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f688c036040 con 0x55c0e0ee7060 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f688c05b050 con 0x55c0e0ee7060 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 10 client.16869 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0e0c25960 con 0x55c0e0ee7060 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c036040 con 0x55c0e0ee7060 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f688c030070 con 0x55c0e0ee7060 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 10 client.16869 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 10 client.16869 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 20 client.16869 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f01c90 0x55c0e0f04080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 4 client.16869 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.574+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0e0e59b90 con 0x55c0e0f01c90 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.575+0000 7f6899428640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f01c90 0x55c0e0f04080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.575+0000 7f6899428640 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f01c90 0x55c0e0f04080 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1ca27d3f6d414e7 server_cookie=3e3483cb5a9ffe6e in_seq=0 out_seq=0 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.575+0000 7f687d7fa640 10 client.16869 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:01.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f687d7fa640 1 -- 172.21.15.78:0/2562702286 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c0e0e59b90 con 0x55c0e0f01c90 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f687d7fa640 10 client.16869 handle_command_reply: tid=0 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 client.16869 shutdown 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 msgr2=0x55c0e0eef170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f688c004480 tx=0x7f688c0093f0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 shutdown_connections 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f01c90 0x55c0e0f04080 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eefbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7980 0x55c0e0eef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f687d7fa640 0 client.16869 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 --2- 172.21.15.78:0/2562702286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7060 0x55c0e0eef170 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 >> 172.21.15.78:0/2562702286 conn(0x55c0e0ee35f0 msgr2=0x55c0e0ee2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 shutdown_connections 2024-09-16T01:24:01.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 1 -- 172.21.15.78:0/2562702286 wait complete. 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.576+0000 7f689e266740 20 client.16869 trim_cache size 0 max 16384 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.592+0000 7f689e266740 1 Processor -- start 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 -- start start 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f68941b18d0 con 0x55c0e0ee7980 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f68941af1a0 con 0x55c0e0ee7060 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689e266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68941ad010 con 0x55c0e0ef2240 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40380/0 (socket says 172.21.15.78:40380) 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 -- 172.21.15.78:0/4132840182 learned_addr learned my addr 172.21.15.78:0/4132840182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:01.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689a42a640 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 -- 172.21.15.78:0/4132840182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 msgr2=0x55c0e0eec3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eec3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 -- 172.21.15.78:0/4132840182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7060 msgr2=0x55c0e0ee9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 -- 172.21.15.78:0/4132840182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0e0ee46a0 con 0x55c0e0ee7980 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f689a42a640 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f6899428640 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f688401afd0 tx=0x7f68840058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5333ae137f06f06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f687f7fe640 1 -- 172.21.15.78:0/4132840182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6884005220 con 0x55c0e0ee7980 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f687f7fe640 1 -- 172.21.15.78:0/4132840182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6884004c80 con 0x55c0e0ee7980 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.593+0000 7f687f7fe640 1 -- 172.21.15.78:0/4132840182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688401a9a0 con 0x55c0e0ee7980 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- 172.21.15.78:0/4132840182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 msgr2=0x55c0e0efd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f688401afd0 tx=0x7f68840058b0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- 172.21.15.78:0/4132840182 shutdown_connections 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ef2240 0x55c0e0eec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0efd8f0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 --2- 172.21.15.78:0/4132840182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ee7060 0x55c0e0ee9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- 172.21.15.78:0/4132840182 >> 172.21.15.78:0/4132840182 conn(0x55c0e0ee35f0 msgr2=0x55c0e0efd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- 172.21.15.78:0/4132840182 shutdown_connections 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- 172.21.15.78:0/4132840182 wait complete. 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 -- start start 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.594+0000 7f689e266740 1 Processor -- start 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ee7060 0x55c0e0f05740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ef2240 0x55c0e0f6de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f68941ad660 con 0x55c0e0ee7980 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f68941aead0 con 0x55c0e0ef2240 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68941add30 con 0x55c0e0ee7060 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 10 client.0 _ll_register_callbacks cb 0x7f688c031a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40392/0 (socket says 172.21.15.78:40392) 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 -- 172.21.15.78:0/597577009 learned_addr learned my addr 172.21.15.78:0/597577009 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689a42a640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ee7060 0x55c0e0f05740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689a42a640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ef2240 0x55c0e0f6de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 -- 172.21.15.78:0/597577009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ee7060 msgr2=0x55c0e0f05740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ee7060 0x55c0e0f05740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 -- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ef2240 msgr2=0x55c0e0f6de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ef2240 0x55c0e0f6de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 -- 172.21.15.78:0/597577009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0e0f6e460 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f6899428640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6884006010 tx=0x7f688400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc96835008b7ba5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f687d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688401a720 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9481+0+0 (secure 0 0 0) 0x7f688401a8c0 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.595+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6884004c80 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f688401ab00 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 10 client.16875 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0e0f75780 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6884004c80 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f688402e7e0 con 0x55c0e0ee7980 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 10 client.16875 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 10 client.16875 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 20 client.16875 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:01.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f75ed0 0x55c0e0f782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:01.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 4 client.16875 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3ee6b2b6", "release": true, "await": true}] 2024-09-16T01:24:01.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3ee6b2b6", "release": true, "await": true}) -- 0x55c0e0e540a0 con 0x55c0e0f75ed0 2024-09-16T01:24:01.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.596+0000 7f689a42a640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f75ed0 0x55c0e0f782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:01.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.597+0000 7f689a42a640 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f75ed0 0x55c0e0f782f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49c342aa3d8f9566 server_cookie=d941443fb2cde9d7 in_seq=0 out_seq=0 2024-09-16T01:24:01.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.597+0000 7f687d7fa640 10 client.16875 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f687d7fa640 1 -- 172.21.15.78:0/597577009 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c0e0e540a0 con 0x55c0e0f75ed0 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f687d7fa640 10 client.16875 handle_command_reply: tid=0 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 client.16875 shutdown 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 msgr2=0x55c0e0f05c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6884006010 tx=0x7f688400be50 comp rx=0 tx=0).stop 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 shutdown_connections 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c0e0f75ed0 0x55c0e0f782f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c0e0ef2240 0x55c0e0f6de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c0e0ee7980 0x55c0e0f05c80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f687d7fa640 0 client.16875 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 --2- 172.21.15.78:0/597577009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0e0ee7060 0x55c0e0f05740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 >> 172.21.15.78:0/597577009 conn(0x55c0e0ee35f0 msgr2=0x55c0e0efda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 shutdown_connections 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 1 -- 172.21.15.78:0/597577009 wait complete. 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.643+0000 7f689e266740 20 client.16875 trim_cache size 0 max 16384 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 48, 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:24:01.647 INFO:teuthology.orchestra.run.smithi078.stdout: "3ee6b2b6": { 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 48, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 131.4, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:24:01.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:01.649 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 msgr2=0x7f68800a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 0x7f68800a58f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55c0e0ee4450 tx=0x7f689000bf60 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 msgr2=0x7f6894198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f689000d950 tx=0x7f689000de10 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 shutdown_connections 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f68800a3430 0x7f68800a58f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6894106a80 0x7f689419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6894101420 0x7f68941988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 --2- 172.21.15.78:0/2179299272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6894100a80 0x7f6894198370 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 >> 172.21.15.78:0/2179299272 conn(0x7f68940fc7a0 msgr2=0x7f6894104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 shutdown_connections 2024-09-16T01:24:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:01.644+0000 7f689be8d640 1 -- 172.21.15.78:0/2179299272 wait complete. 2024-09-16T01:24:02.510 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3ee6b2b6', total seconds elapsed: 14.8 2024-09-16T01:24:03.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:02 smithi078 ceph-mon[30937]: pgmap v415: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-09-16T01:24:03.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:02 smithi193 ceph-mon[35701]: pgmap v415: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-09-16T01:24:03.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:02 smithi148 ceph-mon[35030]: pgmap v415: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-09-16T01:24:05.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:04 smithi078 ceph-mon[30937]: pgmap v416: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-09-16T01:24:05.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:04 smithi193 ceph-mon[35701]: pgmap v416: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-09-16T01:24:05.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:04 smithi148 ceph-mon[35030]: pgmap v416: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 3.3 MiB/s wr, 322 op/s 2024-09-16T01:24:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 770.0 s, 464.9 tps, lat 515.104 ms stddev 1620.117 2024-09-16T01:24:07.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:06 smithi078 ceph-mon[30937]: pgmap v417: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 2.3 MiB/s wr, 247 op/s 2024-09-16T01:24:07.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:06 smithi193 ceph-mon[35701]: pgmap v417: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 2.3 MiB/s wr, 247 op/s 2024-09-16T01:24:07.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:06 smithi148 ceph-mon[35030]: pgmap v417: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 2.3 MiB/s wr, 247 op/s 2024-09-16T01:24:09.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:08 smithi078 ceph-mon[30937]: pgmap v418: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-09-16T01:24:09.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:08 smithi193 ceph-mon[35701]: pgmap v418: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-09-16T01:24:09.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:08 smithi148 ceph-mon[35030]: pgmap v418: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 2.3 MiB/s wr, 215 op/s 2024-09-16T01:24:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.144+0000 7f7ce1559640 1 -- 172.21.15.78:0/3756282678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc101ac0 msgr2=0x7f7cdc101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.144+0000 7f7ce1559640 1 --2- 172.21.15.78:0/3756282678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc101ac0 0x7f7cdc101f20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009930 tx=0x7f7cd002e850 comp rx=0 tx=0).stop 2024-09-16T01:24:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.145+0000 7f7ce1559640 1 -- 172.21.15.78:0/3756282678 shutdown_connections 2024-09-16T01:24:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.145+0000 7f7ce1559640 1 --2- 172.21.15.78:0/3756282678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cdc102460 0x7f7cdc10e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.145+0000 7f7ce1559640 1 --2- 172.21.15.78:0/3756282678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc101ac0 0x7f7cdc101f20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.145+0000 7f7ce1559640 1 --2- 172.21.15.78:0/3756282678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc107eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.145+0000 7f7ce1559640 1 -- 172.21.15.78:0/3756282678 >> 172.21.15.78:0/3756282678 conn(0x7f7cdc0fd820 msgr2=0x7f7cdc0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.148+0000 7f7ce1559640 1 -- 172.21.15.78:0/3756282678 shutdown_connections 2024-09-16T01:24:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.148+0000 7f7ce1559640 1 -- 172.21.15.78:0/3756282678 wait complete. 2024-09-16T01:24:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.148+0000 7f7ce1559640 1 Processor -- start 2024-09-16T01:24:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 -- start start 2024-09-16T01:24:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cdc101ac0 0x7f7cdc1173c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc102460 0x7f7cdc110440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1185f0 con 0x7f7cdc102460 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc118470 con 0x7f7cdc107ad0 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7ce1559640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc118770 con 0x7f7cdc101ac0 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44546/0 (socket says 172.21.15.78:44546) 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.149+0000 7f7cdb7fe640 1 -- 172.21.15.78:0/1834281127 learned_addr learned my addr 172.21.15.78:0/1834281127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 -- 172.21.15.78:0/1834281127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cdc101ac0 msgr2=0x7f7cdc1173c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:24:09.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cdc101ac0 0x7f7cdc1173c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 -- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc102460 msgr2=0x7f7cdc110440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:24:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc102460 0x7f7cdc110440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 -- 172.21.15.78:0/1834281127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc110f90 con 0x7f7cdc107ad0 2024-09-16T01:24:09.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.150+0000 7f7cdb7fe640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400b970 tx=0x7f7cd400be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.152+0000 7f7cc3fff640 1 -- 172.21.15.78:0/1834281127 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4019070 con 0x7f7cdc107ad0 2024-09-16T01:24:09.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.152+0000 7f7cc3fff640 1 -- 172.21.15.78:0/1834281127 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd400cb40 con 0x7f7cdc107ad0 2024-09-16T01:24:09.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.153+0000 7f7cc3fff640 1 -- 172.21.15.78:0/1834281127 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4007650 con 0x7f7cdc107ad0 2024-09-16T01:24:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.154+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc111280 con 0x7f7cdc107ad0 2024-09-16T01:24:09.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.154+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc1151e0 con 0x7f7cdc107ad0 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 Processor -- start 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 -- start start 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec484ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4920a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec3e92f0 con 0x55c9ec48fcb0 2024-09-16T01:24:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec391270 con 0x55c9ec484af0 2024-09-16T01:24:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.158+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec3e52b0 con 0x55c9ec485410 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.160+0000 7f7cda7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.160+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.160+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:55054/0 (socket says 172.21.15.78:55054) 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.160+0000 7f7cda7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4920a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37126/0 (socket says 172.21.15.78:37126) 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.161+0000 7f7cdaffd640 1 -- 172.21.15.78:0/634069639 learned_addr learned my addr 172.21.15.78:0/634069639 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:09.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.161+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec484ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cc3fff640 1 -- 172.21.15.78:0/1834281127 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7cd400ccf0 con 0x7f7cdc107ad0 2024-09-16T01:24:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cdaffd640 1 -- 172.21.15.78:0/634069639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 msgr2=0x55c9ec484ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec484ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cdaffd640 1 -- 172.21.15.78:0/634069639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 msgr2=0x55c9ec4920a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4920a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cdaffd640 1 -- 172.21.15.78:0/634069639 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9ec390340 con 0x55c9ec485410 2024-09-16T01:24:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.163+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec484ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:24:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.164+0000 7f7cc3fff640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 0x7f7cc40a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.164+0000 7f7cc3fff640 1 -- 172.21.15.78:0/1834281127 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f7cd40cf4b0 con 0x7f7cdc107ad0 2024-09-16T01:24:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.165+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009770 tx=0x7f7ccc009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=601c72b056eae1b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.166+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 0x7f7cc40a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.167+0000 7f7cc17fa640 1 -- 172.21.15.78:0/634069639 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc015020 con 0x55c9ec485410 2024-09-16T01:24:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.167+0000 7f7cc17fa640 1 -- 172.21.15.78:0/634069639 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ccc011420 con 0x55c9ec485410 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.168+0000 7f7ce43d1740 1 -- 172.21.15.78:0/634069639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 msgr2=0x55c9ec48f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.168+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009770 tx=0x7f7ccc009c30 comp rx=0 tx=0).stop 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 0x7f7cc40a5a20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005c20 tx=0x7f7cd0005a20 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7ce43d1740 1 -- 172.21.15.78:0/634069639 shutdown_connections 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4920a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48f770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/634069639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec484ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.170 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.169+0000 7f7ce43d1740 1 -- 172.21.15.78:0/634069639 >> 172.21.15.78:0/634069639 conn(0x55c9ec481060 msgr2=0x55c9ec481440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.174+0000 7f7ce43d1740 1 -- 172.21.15.78:0/634069639 shutdown_connections 2024-09-16T01:24:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.175+0000 7f7ce43d1740 1 -- 172.21.15.78:0/634069639 wait complete. 2024-09-16T01:24:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.175+0000 7f7ce43d1740 1 -- start start 2024-09-16T01:24:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.176+0000 7f7ce43d1740 1 Processor -- start 2024-09-16T01:24:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec48fcb0 0x55c9ec48d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec4928d0 con 0x55c9ec484af0 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec372290 con 0x55c9ec48fcb0 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9ec492a50 con 0x55c9ec485410 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.178+0000 7f7ce43d1740 10 client.0 _ll_register_callbacks cb 0x55c9ec3a3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.181+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.181+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37136/0 (socket says 172.21.15.78:37136) 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.181+0000 7f7cdaffd640 1 -- 172.21.15.78:0/2774997481 learned_addr learned my addr 172.21.15.78:0/2774997481 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.181+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.181+0000 7f7cdb7fe640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec48fcb0 0x55c9ec48d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 -- 172.21.15.78:0/2774997481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 msgr2=0x55c9ec48d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 -- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec48fcb0 msgr2=0x55c9ec48d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec48fcb0 0x55c9ec48d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 -- 172.21.15.78:0/2774997481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9ec0fc210 con 0x55c9ec484af0 2024-09-16T01:24:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc011f00 tx=0x7f7ccc00bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2dab0e37bd554e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.182+0000 7f7cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.186+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc004240 con 0x55c9ec484af0 2024-09-16T01:24:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.186+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ccc011420 con 0x55c9ec484af0 2024-09-16T01:24:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.186+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f7ccc0043e0 con 0x55c9ec484af0 2024-09-16T01:24:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.188+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7ccc010840 con 0x55c9ec484af0 2024-09-16T01:24:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.188+0000 7f7ce43d1740 10 client.16884 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.188+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c9ec1c8620 con 0x55c9ec484af0 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.188+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ccc011420 con 0x55c9ec484af0 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7ccc011a80 con 0x55c9ec484af0 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 10 client.16884 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 10 client.16884 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 20 client.16884 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec49ffa0 0x55c9ec4a2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 4 client.16884 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.189+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9ec3f6a20 con 0x55c9ec49ffa0 2024-09-16T01:24:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.191+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec49ffa0 0x55c9ec4a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.197+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec49ffa0 0x55c9ec4a2390 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c93524205d7251 server_cookie=53acee5f2df8e0e3 in_seq=0 out_seq=0 2024-09-16T01:24:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.197+0000 7f7cbeff5640 10 client.16884 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.201+0000 7f7cbeff5640 1 -- 172.21.15.78:0/2774997481 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9ec3f6a20 con 0x55c9ec49ffa0 2024-09-16T01:24:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.201+0000 7f7cbeff5640 10 client.16884 handle_command_reply: tid=0 2024-09-16T01:24:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.201+0000 7f7ce43d1740 1 client.16884 shutdown 2024-09-16T01:24:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.203+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 msgr2=0x55c9ec48cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.203+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc011f00 tx=0x7f7ccc00bb00 comp rx=0 tx=0).stop 2024-09-16T01:24:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 shutdown_connections 2024-09-16T01:24:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec49ffa0 0x55c9ec4a2390 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec48fcb0 0x55c9ec48d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec485410 0x55c9ec48d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/2774997481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48cb30 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 >> 172.21.15.78:0/2774997481 conn(0x55c9ec481060 msgr2=0x55c9ec4801c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.206+0000 7f7cbeff5640 0 client.16884 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.208+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 shutdown_connections 2024-09-16T01:24:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.210+0000 7f7ce43d1740 1 -- 172.21.15.78:0/2774997481 wait complete. 2024-09-16T01:24:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.215+0000 7f7ce43d1740 20 client.16884 trim_cache size 0 max 16384 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.244+0000 7f7ce43d1740 1 Processor -- start 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.244+0000 7f7ce43d1740 1 -- start start 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.244+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.244+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 0x55c9ec486c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.245+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.245+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1c9090 con 0x55c9ec484af0 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.245+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1c6960 con 0x55c9ec485410 2024-09-16T01:24:09.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.245+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc103200 con 0x55c9ec48fcb0 2024-09-16T01:24:09.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.246+0000 7f7cda7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 0x55c9ec486c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.249+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.249+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4a7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:55066/0 (socket says 172.21.15.78:55066) 2024-09-16T01:24:09.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.249+0000 7f7cdb7fe640 1 -- 172.21.15.78:0/3114766493 learned_addr learned my addr 172.21.15.78:0/3114766493 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.249+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3114766493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 msgr2=0x55c9ec4a7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4a7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3114766493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 msgr2=0x55c9ec48d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3114766493 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9ec48b060 con 0x55c9ec485410 2024-09-16T01:24:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.250+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 0x55c9ec486c60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd000e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d9c973390b0f66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.254+0000 7f7cc0ff9640 1 -- 172.21.15.78:0/3114766493 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0049030 con 0x55c9ec485410 2024-09-16T01:24:09.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.254+0000 7f7cc0ff9640 1 -- 172.21.15.78:0/3114766493 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd004a040 con 0x55c9ec485410 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.254+0000 7f7cc0ff9640 1 -- 172.21.15.78:0/3114766493 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00436c0 con 0x55c9ec485410 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.255+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3114766493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 msgr2=0x55c9ec486c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.255+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 0x55c9ec486c60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd000e000 comp rx=0 tx=0).stop 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.256+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3114766493 shutdown_connections 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.256+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec4a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.256+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec485410 0x55c9ec486c60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.256+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3114766493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec484af0 0x55c9ec48d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.256+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3114766493 >> 172.21.15.78:0/3114766493 conn(0x55c9ec481060 msgr2=0x55c9ec486550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.263+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3114766493 shutdown_connections 2024-09-16T01:24:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3114766493 wait complete. 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 -- start start 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 Processor -- start 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec492dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 0x55c9ec493310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec508c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc115cb0 con 0x55c9ec485410 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1c6290 con 0x55c9ec484af0 2024-09-16T01:24:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.267+0000 7f7ce43d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc073e30 con 0x55c9ec48fcb0 2024-09-16T01:24:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.268+0000 7f7ce43d1740 10 client.0 _ll_register_callbacks cb 0x7f7ccc0095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:09.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.268+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec492dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.268+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec492dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:44568/0 (socket says 172.21.15.78:44568) 2024-09-16T01:24:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.268+0000 7f7cdaffd640 1 -- 172.21.15.78:0/3136864008 learned_addr learned my addr 172.21.15.78:0/3136864008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.269+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 0x55c9ec493310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.270+0000 7f7cdb7fe640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec508c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.271+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3136864008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 msgr2=0x55c9ec508c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.271+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec508c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.271+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 msgr2=0x55c9ec492dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.271+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec492dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.271+0000 7f7cda7fc640 1 -- 172.21.15.78:0/3136864008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9ec493bc0 con 0x55c9ec485410 2024-09-16T01:24:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.274+0000 7f7cda7fc640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 0x55c9ec493310 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd0035c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8200109c1c86f1fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:09.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.275+0000 7f7cbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.275+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0049040 con 0x55c9ec485410 2024-09-16T01:24:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.275+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd004a040 con 0x55c9ec485410 2024-09-16T01:24:09.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.276+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f7cd00357b0 con 0x55c9ec485410 2024-09-16T01:24:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.280+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7cd0068050 con 0x55c9ec485410 2024-09-16T01:24:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.280+0000 7f7ce43d1740 10 client.16890 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.280+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c9ec50a290 con 0x55c9ec485410 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd004a040 con 0x55c9ec485410 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7cd000f040 con 0x55c9ec485410 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 10 client.16890 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 10 client.16890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 20 client.16890 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec50a930 0x55c9ec50cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 4 client.16890 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:24:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.286+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9ec3f0d70 con 0x55c9ec50a930 2024-09-16T01:24:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.288+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec50a930 0x55c9ec50cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.298+0000 7f7cdaffd640 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec50a930 0x55c9ec50cd50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e1a41c3e340dd4 server_cookie=64500139efc2b7e8 in_seq=0 out_seq=0 2024-09-16T01:24:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.299+0000 7f7cbeff5640 10 client.16890 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7cbeff5640 1 -- 172.21.15.78:0/3136864008 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9ec3f0d70 con 0x55c9ec50a930 2024-09-16T01:24:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7cbeff5640 10 client.16890 handle_command_reply: tid=0 2024-09-16T01:24:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 client.16890 shutdown 2024-09-16T01:24:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 msgr2=0x55c9ec493310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 0x55c9ec493310 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd0035c10 comp rx=0 tx=0).stop 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 shutdown_connections 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55c9ec50a930 0x55c9ec50cd50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9ec48fcb0 0x55c9ec508c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c9ec485410 0x55c9ec493310 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 --2- 172.21.15.78:0/3136864008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c9ec484af0 0x55c9ec492dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 >> 172.21.15.78:0/3136864008 conn(0x55c9ec481060 msgr2=0x55c9ec4801c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7cbeff5640 0 client.16890 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 shutdown_connections 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 1 -- 172.21.15.78:0/3136864008 wait complete. 2024-09-16T01:24:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.550+0000 7f7ce43d1740 20 client.16890 trim_cache size 0 max 16384 2024-09-16T01:24:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:24:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:24:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:24:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 msgr2=0x7f7cc40a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 0x7f7cc40a5a20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005c20 tx=0x7f7cd0005a20 comp rx=0 tx=0).stop 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 msgr2=0x7f7cdc110980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400b970 tx=0x7f7cd400be30 comp rx=0 tx=0).stop 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 shutdown_connections 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7cc40a3560 0x7f7cc40a5a20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cdc107ad0 0x7f7cdc110980 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cdc102460 0x7f7cdc110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 --2- 172.21.15.78:0/1834281127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cdc101ac0 0x7f7cdc1173c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.551+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 >> 172.21.15.78:0/1834281127 conn(0x7f7cdc0fd820 msgr2=0x7f7cdc0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.552+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 shutdown_connections 2024-09-16T01:24:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:09.552+0000 7f7ce1559640 1 -- 172.21.15.78:0/1834281127 wait complete. 2024-09-16T01:24:10.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:09 smithi193 ceph-mon[35701]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T01:24:10.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:09 smithi148 ceph-mon[35030]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T01:24:10.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:09 smithi078 ceph-mon[30937]: osdmap e110: 12 total, 12 up, 12 in 2024-09-16T01:24:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 775.0 s, 499.2 tps, lat 203.917 ms stddev 284.347 2024-09-16T01:24:11.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:10 smithi148 ceph-mon[35030]: pgmap v420: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-09-16T01:24:11.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:10 smithi078 ceph-mon[30937]: pgmap v420: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-09-16T01:24:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:10 smithi193 ceph-mon[35701]: pgmap v420: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-09-16T01:24:12.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:11 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:24:12.265 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:11 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:24:12.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:11 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:24:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:12 smithi148 ceph-mon[35030]: pgmap v421: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-09-16T01:24:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:24:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:24:13.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:24:13.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:12 smithi078 ceph-mon[30937]: pgmap v421: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-09-16T01:24:13.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:24:13.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:24:13.209 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:24:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:12 smithi193 ceph-mon[35701]: pgmap v421: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.9 MiB/s wr, 404 op/s 2024-09-16T01:24:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:24:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:24:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:24:15.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:14 smithi148 ceph-mon[35030]: pgmap v422: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.2 MiB/s wr, 440 op/s 2024-09-16T01:24:15.140 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:14 smithi078 ceph-mon[30937]: pgmap v422: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.2 MiB/s wr, 440 op/s 2024-09-16T01:24:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:14 smithi193 ceph-mon[35701]: pgmap v422: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 4.2 MiB/s wr, 440 op/s 2024-09-16T01:24:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 780.0 s, 545.2 tps, lat 185.499 ms stddev 283.921 2024-09-16T01:24:17.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:16 smithi148 ceph-mon[35030]: pgmap v423: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 5.3 MiB/s wr, 510 op/s 2024-09-16T01:24:17.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:16 smithi078 ceph-mon[30937]: pgmap v423: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 5.3 MiB/s wr, 510 op/s 2024-09-16T01:24:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:16 smithi193 ceph-mon[35701]: pgmap v423: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 5.3 MiB/s wr, 510 op/s 2024-09-16T01:24:19.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:18 smithi148 ceph-mon[35030]: pgmap v424: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:24:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:18 smithi078 ceph-mon[30937]: pgmap v424: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:24:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:18 smithi193 ceph-mon[35701]: pgmap v424: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 5.3 MiB/s wr, 520 op/s 2024-09-16T01:24:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 785.0 s, 525.4 tps, lat 189.759 ms stddev 211.957 2024-09-16T01:24:21.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:20 smithi148 ceph-mon[35030]: pgmap v425: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 5.3 MiB/s wr, 524 op/s 2024-09-16T01:24:21.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:20 smithi148 ceph-mon[35030]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T01:24:21.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:20 smithi078 ceph-mon[30937]: pgmap v425: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 5.3 MiB/s wr, 524 op/s 2024-09-16T01:24:21.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:20 smithi078 ceph-mon[30937]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T01:24:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:20 smithi193 ceph-mon[35701]: pgmap v425: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 5.3 MiB/s wr, 524 op/s 2024-09-16T01:24:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:20 smithi193 ceph-mon[35701]: osdmap e111: 12 total, 12 up, 12 in 2024-09-16T01:24:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 -- 172.21.15.78:0/900180281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9011b140 msgr2=0x7f7f9011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/900180281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9011b140 0x7f7f9011d530 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009ca0 tx=0x7f7f8802a8f0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 -- 172.21.15.78:0/900180281 shutdown_connections 2024-09-16T01:24:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/900180281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9011b140 0x7f7f9011d530 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/900180281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f901095d0 0x7f7f90109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/900180281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f901091f0 0x7f7f90110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.711+0000 7f7f96f4b640 1 -- 172.21.15.78:0/900180281 >> 172.21.15.78:0/900180281 conn(0x7f7f9006ce00 msgr2=0x7f7f9006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.716+0000 7f7f96f4b640 1 -- 172.21.15.78:0/900180281 shutdown_connections 2024-09-16T01:24:22.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.719+0000 7f7f96f4b640 1 -- 172.21.15.78:0/900180281 wait complete. 2024-09-16T01:24:22.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.719+0000 7f7f96f4b640 1 Processor -- start 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 -- start start 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f901091f0 0x7f7f9011a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f9011b140 0x7f7f90114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f90111370 con 0x7f7f901095d0 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f901111f0 con 0x7f7f901091f0 2024-09-16T01:24:22.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.724+0000 7f7f96f4b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f901114f0 con 0x7f7f9011b140 2024-09-16T01:24:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.725+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.726+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34424/0 (socket says 172.21.15.78:34424) 2024-09-16T01:24:22.727 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.726+0000 7f7f87fff640 1 -- 172.21.15.78:0/611338737 learned_addr learned my addr 172.21.15.78:0/611338737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.727+0000 7f7f87fff640 1 -- 172.21.15.78:0/611338737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f9011b140 msgr2=0x7f7f90114e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:24:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.727+0000 7f7f87fff640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f9011b140 0x7f7f90114e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.727+0000 7f7f87fff640 1 -- 172.21.15.78:0/611338737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f901091f0 msgr2=0x7f7f9011a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:24:22.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.727+0000 7f7f87fff640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f901091f0 0x7f7f9011a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.727+0000 7f7f87fff640 1 -- 172.21.15.78:0/611338737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f901155b0 con 0x7f7f901095d0 2024-09-16T01:24:22.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.728+0000 7f7f87fff640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00b450 tx=0x7f7f8c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.729+0000 7f7f85ffb640 1 -- 172.21.15.78:0/611338737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c015020 con 0x7f7f901095d0 2024-09-16T01:24:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.729+0000 7f7f85ffb640 1 -- 172.21.15.78:0/611338737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c011d70 con 0x7f7f901095d0 2024-09-16T01:24:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.729+0000 7f7f85ffb640 1 -- 172.21.15.78:0/611338737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00f340 con 0x7f7f901095d0 2024-09-16T01:24:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.729+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f901bc0a0 con 0x7f7f901095d0 2024-09-16T01:24:22.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.729+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f901bc510 con 0x7f7f901095d0 2024-09-16T01:24:22.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.737+0000 7f7f99922740 1 Processor -- start 2024-09-16T01:24:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.738+0000 7f7f85ffb640 1 -- 172.21.15.78:0/611338737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7f8c018020 con 0x7f7f901095d0 2024-09-16T01:24:22.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 -- start start 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1d5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d6410 0x560f7b1e0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560f7b13a2f0 con 0x560f7b1d6410 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560f7b0e2270 con 0x560f7b1e0cb0 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f99922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f7b1362b0 con 0x560f7b1d5af0 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:53958/0 (socket says 172.21.15.78:53958) 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.739+0000 7f7f87fff640 1 -- 172.21.15.78:0/2661181984 learned_addr learned my addr 172.21.15.78:0/2661181984 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 -- 172.21.15.78:0/2661181984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 msgr2=0x560f7b1d5ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1d5ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 -- 172.21.15.78:0/2661181984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d6410 msgr2=0x560f7b1e0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d6410 0x560f7b1e0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 -- 172.21.15.78:0/2661181984 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f7b0e1340 con 0x560f7b1e0cb0 2024-09-16T01:24:22.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0cccd0 tx=0x7f7f8c0d7750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3711ee7698364d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.740+0000 7f7f85ffb640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 0x7f7f780a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.741+0000 7f7f85ffb640 1 -- 172.21.15.78:0/611338737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f7f8c0cf980 con 0x7f7f901095d0 2024-09-16T01:24:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.741+0000 7f7f76ffd640 1 -- 172.21.15.78:0/2661181984 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c0e1670 con 0x560f7b1e0cb0 2024-09-16T01:24:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.741+0000 7f7f76ffd640 1 -- 172.21.15.78:0/2661181984 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c0dc070 con 0x560f7b1e0cb0 2024-09-16T01:24:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.741+0000 7f7f76ffd640 1 -- 172.21.15.78:0/2661181984 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c0d8d00 con 0x560f7b1e0cb0 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f954e8640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 0x7f7f780a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 -- 172.21.15.78:0/2661181984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 msgr2=0x560f7b1e30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0cccd0 tx=0x7f7f8c0d7750 comp rx=0 tx=0).stop 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 -- 172.21.15.78:0/2661181984 shutdown_connections 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1e30a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d6410 0x560f7b1e0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 --2- 172.21.15.78:0/2661181984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1d5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.742+0000 7f7f99922740 1 -- 172.21.15.78:0/2661181984 >> 172.21.15.78:0/2661181984 conn(0x560f7b1d2060 msgr2=0x560f7b1d2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.743+0000 7f7f954e8640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 0x7f7f780a5b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f8802aee0 tx=0x7f7f88005f40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.750+0000 7f7f99922740 1 -- 172.21.15.78:0/2661181984 shutdown_connections 2024-09-16T01:24:22.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.754+0000 7f7f99922740 1 -- 172.21.15.78:0/2661181984 wait complete. 2024-09-16T01:24:22.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.754+0000 7f7f99922740 1 -- start start 2024-09-16T01:24:22.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.754+0000 7f7f99922740 1 Processor -- start 2024-09-16T01:24:22.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1de260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x560f7b1e3880 con 0x560f7b1de7a0 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560f7b0c3290 con 0x560f7b1e0cb0 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.755+0000 7f7f99922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f7b1e3a00 con 0x560f7b1d5af0 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.756+0000 7f7f99922740 10 client.0 _ll_register_callbacks cb 0x560f7b0f4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.757+0000 7f7f954e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.757+0000 7f7f954e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34442/0 (socket says 172.21.15.78:34442) 2024-09-16T01:24:22.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.757+0000 7f7f954e8640 1 -- 172.21.15.78:0/371776222 learned_addr learned my addr 172.21.15.78:0/371776222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 -- 172.21.15.78:0/371776222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 msgr2=0x560f7b1ddd20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1ddd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 -- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 msgr2=0x560f7b1de260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1de260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 -- 172.21.15.78:0/371776222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f7ae4d210 con 0x560f7b1de7a0 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f954e8640 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7f88003c20 tx=0x7f7f880095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c15c55c824b29c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.759+0000 7f7f6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.760+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88003e50 con 0x560f7b1de7a0 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.760+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f8c0dc070 con 0x560f7b1de7a0 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.760+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f7f880045c0 con 0x560f7b1de7a0 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.761+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f88058050 con 0x560f7b1de7a0 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.761+0000 7f7f99922740 10 client.16899 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.761+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f7af19620 con 0x560f7b1de7a0 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c0dc070 con 0x560f7b1de7a0 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7f88030860 con 0x560f7b1de7a0 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 10 client.16899 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 20 client.16899 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b1f1700 0x560f7b1f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 4 client.16899 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.762+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f7b147a20 con 0x560f7b1f1700 2024-09-16T01:24:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.766+0000 7f7f94ce7640 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b1f1700 0x560f7b1f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.773 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.772+0000 7f7f94ce7640 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b1f1700 0x560f7b1f3bc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd5689eea3bf6f7 server_cookie=e6b12506d52ac625 in_seq=0 out_seq=0 2024-09-16T01:24:22.774 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.772+0000 7f7f6ffff640 10 client.16899 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:22.775 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.773+0000 7f7f6ffff640 1 -- 172.21.15.78:0/371776222 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f7b147a20 con 0x560f7b1f1700 2024-09-16T01:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.774+0000 7f7f6ffff640 10 client.16899 handle_command_reply: tid=0 2024-09-16T01:24:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.774+0000 7f7f99922740 1 client.16899 shutdown 2024-09-16T01:24:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.777+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 msgr2=0x560f7b1ecdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.777+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7f88003c20 tx=0x7f7f880095c0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 shutdown_connections 2024-09-16T01:24:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b1f1700 0x560f7b1f3bc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1ecdc0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1e0cb0 0x560f7b1de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 --2- 172.21.15.78:0/371776222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1d5af0 0x560f7b1ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 >> 172.21.15.78:0/371776222 conn(0x560f7b1d2060 msgr2=0x560f7b1d7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.779+0000 7f7f6ffff640 0 client.16899 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.780+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 shutdown_connections 2024-09-16T01:24:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.781+0000 7f7f99922740 1 -- 172.21.15.78:0/371776222 wait complete. 2024-09-16T01:24:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.782+0000 7f7f99922740 20 client.16899 trim_cache size 0 max 16384 2024-09-16T01:24:22.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.813+0000 7f7f99922740 1 Processor -- start 2024-09-16T01:24:22.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.814+0000 7f7f99922740 1 -- start start 2024-09-16T01:24:22.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.814+0000 7f7f99922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d5af0 0x560f7b1ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.814+0000 7f7f99922740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f99922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f99922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0129f0 con 0x560f7b1d5af0 2024-09-16T01:24:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:53978/0 (socket says 172.21.15.78:53978) 2024-09-16T01:24:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f87fff640 1 -- 172.21.15.78:0/495500914 learned_addr learned my addr 172.21.15.78:0/495500914 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f99922740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0102c0 con 0x560f7b1de7a0 2024-09-16T01:24:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.815+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00e130 con 0x560f7b1e0cb0 2024-09-16T01:24:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.820+0000 7f7f87fff640 1 -- 172.21.15.78:0/495500914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 msgr2=0x560f7b1f94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.820+0000 7f7f87fff640 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.820+0000 7f7f87fff640 1 -- 172.21.15.78:0/495500914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d5af0 msgr2=0x560f7b1ecbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.820+0000 7f7f87fff640 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d5af0 0x560f7b1ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.820+0000 7f7f87fff640 1 -- 172.21.15.78:0/495500914 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f7b1d5920 con 0x560f7b1de7a0 2024-09-16T01:24:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.823+0000 7f7f87fff640 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00be10 tx=0x7f7f8c0d5340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a074e4a5b2cd7a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.828+0000 7f7f767fc640 1 -- 172.21.15.78:0/495500914 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c09b6f0 con 0x560f7b1de7a0 2024-09-16T01:24:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.828+0000 7f7f767fc640 1 -- 172.21.15.78:0/495500914 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c0dc040 con 0x560f7b1de7a0 2024-09-16T01:24:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.829+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 msgr2=0x560f7b1d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.829+0000 7f7f99922740 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00be10 tx=0x7f7f8c0d5340 comp rx=0 tx=0).stop 2024-09-16T01:24:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.835+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 shutdown_connections 2024-09-16T01:24:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.835+0000 7f7f99922740 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1f94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.835+0000 7f7f99922740 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1de7a0 0x560f7b1d7cb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.835+0000 7f7f99922740 1 --2- 172.21.15.78:0/495500914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1d5af0 0x560f7b1ecbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.835+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 >> 172.21.15.78:0/495500914 conn(0x560f7b1d2060 msgr2=0x560f7b1d75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.841+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 shutdown_connections 2024-09-16T01:24:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.841+0000 7f7f99922740 1 -- 172.21.15.78:0/495500914 wait complete. 2024-09-16T01:24:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.841+0000 7f7f99922740 1 -- start start 2024-09-16T01:24:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.841+0000 7f7f99922740 1 Processor -- start 2024-09-16T01:24:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1d5af0 0x560f7b1e8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00e780 con 0x560f7b1de7a0 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00fbf0 con 0x560f7b1d5af0 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c00ee50 con 0x560f7b1e0cb0 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.845+0000 7f7f99922740 10 client.0 _ll_register_callbacks cb 0x7f7f88009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34462/0 (socket says 172.21.15.78:34462) 2024-09-16T01:24:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 -- 172.21.15.78:0/2084037440 learned_addr learned my addr 172.21.15.78:0/2084037440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f94ce7640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1d5af0 0x560f7b1e8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 -- 172.21.15.78:0/2084037440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 msgr2=0x560f7b1da150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1da150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 -- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1d5af0 msgr2=0x560f7b1e8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1d5af0 0x560f7b1e8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.846+0000 7f7f87fff640 1 -- 172.21.15.78:0/2084037440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f7b1da690 con 0x560f7b1de7a0 2024-09-16T01:24:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.852+0000 7f7f87fff640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0982f0 tx=0x7f7f8c0d5660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef262526a60d3812 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.852+0000 7f7f6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.858+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c09b460 con 0x560f7b1de7a0 2024-09-16T01:24:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.858+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f7f8c09bb80 con 0x560f7b1de7a0 2024-09-16T01:24:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.858+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f8c0dc040 con 0x560f7b1de7a0 2024-09-16T01:24:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.860+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f8c09b6f0 con 0x560f7b1de7a0 2024-09-16T01:24:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.860+0000 7f7f99922740 10 client.16902 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.860+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f7b25bc50 con 0x560f7b1de7a0 2024-09-16T01:24:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f8c0dc040 con 0x560f7b1de7a0 2024-09-16T01:24:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f7f8c0ba0d0 con 0x560f7b1de7a0 2024-09-16T01:24:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 10 client.16902 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 10 client.16902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 20 client.16902 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b25c1f0 0x560f7b25e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 4 client.16902 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:24:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.861+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f7b141d70 con 0x560f7b25c1f0 2024-09-16T01:24:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.866+0000 7f7f94ce7640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b25c1f0 0x560f7b25e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:22.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.870+0000 7f7f94ce7640 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b25c1f0 0x560f7b25e610 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9c6857d29b44878 server_cookie=baf05bb9086cfb6b in_seq=0 out_seq=0 2024-09-16T01:24:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.871+0000 7f7f6ffff640 10 client.16902 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.919+0000 7f7f6ffff640 1 -- 172.21.15.78:0/2084037440 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f7b141d70 con 0x560f7b25c1f0 2024-09-16T01:24:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.919+0000 7f7f6ffff640 10 client.16902 handle_command_reply: tid=0 2024-09-16T01:24:22.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.919+0000 7f7f99922740 1 client.16902 shutdown 2024-09-16T01:24:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.925+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 msgr2=0x560f7b1e9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.925+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0982f0 tx=0x7f7f8c0d5660 comp rx=0 tx=0).stop 2024-09-16T01:24:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 shutdown_connections 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x560f7b25c1f0 0x560f7b25e610 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f7b1e0cb0 0x560f7b1da150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f7b1de7a0 0x560f7b1e9260 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 --2- 172.21.15.78:0/2084037440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f7b1d5af0 0x560f7b1e8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 >> 172.21.15.78:0/2084037440 conn(0x560f7b1d2060 msgr2=0x560f7b1dfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f6ffff640 0 client.16902 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 shutdown_connections 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 1 -- 172.21.15.78:0/2084037440 wait complete. 2024-09-16T01:24:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.926+0000 7f7f99922740 20 client.16902 trim_cache size 0 max 16384 2024-09-16T01:24:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:24:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:24:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:24:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 msgr2=0x7f7f780a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 0x7f7f780a5b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f8802aee0 tx=0x7f7f88005f40 comp rx=0 tx=0).stop 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 msgr2=0x7f7f90114920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00b450 tx=0x7f7f8c00b910 comp rx=0 tx=0).stop 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 shutdown_connections 2024-09-16T01:24:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7f780a3690 0x7f7f780a5b50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f9011b140 0x7f7f90114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f901095d0 0x7f7f90114920 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 --2- 172.21.15.78:0/611338737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f901091f0 0x7f7f9011a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.927+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 >> 172.21.15.78:0/611338737 conn(0x7f7f9006ce00 msgr2=0x7f7f90071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.928+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 shutdown_connections 2024-09-16T01:24:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:22.928+0000 7f7f96f4b640 1 -- 172.21.15.78:0/611338737 wait complete. 2024-09-16T01:24:23.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:22 smithi148 ceph-mon[35030]: pgmap v427: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 5.4 MiB/s wr, 534 op/s 2024-09-16T01:24:23.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:22 smithi078 ceph-mon[30937]: pgmap v427: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 5.4 MiB/s wr, 534 op/s 2024-09-16T01:24:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:22 smithi193 ceph-mon[35701]: pgmap v427: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 5.4 MiB/s wr, 534 op/s 2024-09-16T01:24:23.682 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-09-16T01:24:23.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:24:24.200 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:24:24.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 -- 172.21.15.78:0/2059855464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0100a80 msgr2=0x7fabc0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:24.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 --2- 172.21.15.78:0/2059855464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0100a80 0x7fabc0100ee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009900 tx=0x7fabb002a410 comp rx=0 tx=0).stop 2024-09-16T01:24:24.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 -- 172.21.15.78:0/2059855464 shutdown_connections 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 --2- 172.21.15.78:0/2059855464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 0x7fabc010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 --2- 172.21.15.78:0/2059855464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0100a80 0x7fabc0100ee0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 --2- 172.21.15.78:0/2059855464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0106a80 0x7fabc0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 -- 172.21.15.78:0/2059855464 >> 172.21.15.78:0/2059855464 conn(0x7fabc00fc7a0 msgr2=0x7fabc00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 -- 172.21.15.78:0/2059855464 shutdown_connections 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 -- 172.21.15.78:0/2059855464 wait complete. 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.792+0000 7fabc874f640 1 Processor -- start 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 -- start start 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 0x7fabc01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:24.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc5cea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 0x7fabc01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 0x7fabc019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0110e90 con 0x7fabc0101420 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0110d10 con 0x7fabc0106a80 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc874f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0111010 con 0x7fabc0100a80 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc64eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc64eb640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:43658/0 (socket says 172.21.15.78:43658) 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc64eb640 1 -- 172.21.15.78:0/2613143658 learned_addr learned my addr 172.21.15.78:0/2613143658 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.793+0000 7fabc6cec640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 0x7fabc019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 -- 172.21.15.78:0/2613143658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 msgr2=0x7fabc019cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 0x7fabc019cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 -- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 msgr2=0x7fabc01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 0x7fabc01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:24.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 -- 172.21.15.78:0/2613143658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc019d240 con 0x7fabc0100a80 2024-09-16T01:24:24.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc6cec640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 0x7fabc019cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:24:24.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabc64eb640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00e9f0 tx=0x7fabbc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:24.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00cd80 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabbc010910 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.794+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc018990 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.795+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc019d530 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.796+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc01a4e10 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.797+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc01021c0 con 0x7fabc0100a80 2024-09-16T01:24:24.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.798+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fabbc010ac0 con 0x7fabc0100a80 2024-09-16T01:24:24.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.799+0000 7fabb77fe640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 0x7fabac0a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:24.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.799+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7fabbc0d08a0 con 0x7fabc0100a80 2024-09-16T01:24:24.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.799+0000 7fabc5cea640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 0x7fabac0a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:24.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.799+0000 7fabc5cea640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 0x7fabac0a59c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fabb0004650 tx=0x7fabb0000bc0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:24.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:24.803+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fabbc09ceb0 con 0x7fabc0100a80 2024-09-16T01:24:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:24 smithi078 ceph-mon[30937]: pgmap v428: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 5.8 MiB/s wr, 554 op/s 2024-09-16T01:24:25.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:24 smithi148 ceph-mon[35030]: pgmap v428: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 5.8 MiB/s wr, 554 op/s 2024-09-16T01:24:25.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.187+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabc01995f0 con 0x7fabc0100a80 2024-09-16T01:24:25.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.188+0000 7fabb77fe640 1 -- 172.21.15.78:0/2613143658 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fabbc09c7b0 con 0x7fabc0100a80 2024-09-16T01:24:25.190 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:24:25.190 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:24:25.191 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:24:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 msgr2=0x7fabac0a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 0x7fabac0a59c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fabb0004650 tx=0x7fabb0000bc0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 msgr2=0x7fabc0198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00e9f0 tx=0x7fabbc00eeb0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 shutdown_connections 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fabac0a3500 0x7fabac0a59c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabc0106a80 0x7fabc019cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc0101420 0x7fabc01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 --2- 172.21.15.78:0/2613143658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0100a80 0x7fabc0198290 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00e9f0 tx=0x7fabbc00eeb0 comp rx=0 tx=0).stop 2024-09-16T01:24:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.195+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 >> 172.21.15.78:0/2613143658 conn(0x7fabc00fc7a0 msgr2=0x7fabc006a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:25.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.196+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 shutdown_connections 2024-09-16T01:24:25.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:25.196+0000 7fabc874f640 1 -- 172.21.15.78:0/2613143658 wait complete. 2024-09-16T01:24:25.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:24 smithi193 ceph-mon[35701]: pgmap v428: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 5.8 MiB/s wr, 554 op/s 2024-09-16T01:24:25.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 790.0 s, 481.0 tps, lat 199.458 ms stddev 231.687 2024-09-16T01:24:26.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:25 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2613143658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:26.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:25 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2613143658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:26.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:25 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2613143658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:26 smithi148 ceph-mon[35030]: pgmap v429: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-09-16T01:24:27.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:26 smithi078 ceph-mon[30937]: pgmap v429: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-09-16T01:24:27.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:26 smithi193 ceph-mon[35701]: pgmap v429: 129 pgs: 129 active+clean; 2.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-09-16T01:24:28.065 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0' 2024-09-16T01:24:28.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0 2024-09-16T01:24:28.469 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:24:29.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:28 smithi148 ceph-mon[35030]: pgmap v430: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.7 MiB/s wr, 518 op/s 2024-09-16T01:24:29.179 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:28 smithi078 ceph-mon[30937]: pgmap v430: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.7 MiB/s wr, 518 op/s 2024-09-16T01:24:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:28 smithi193 ceph-mon[35701]: pgmap v430: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.7 MiB/s wr, 518 op/s 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.516+0000 7efcc551a640 1 -- 172.21.15.78:0/3677077702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc011b140 msgr2=0x7efcc011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.516+0000 7efcc551a640 1 --2- 172.21.15.78:0/3677077702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc011b140 0x7efcc011d530 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009860 tx=0x7efcb002a380 comp rx=0 tx=0).stop 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.517+0000 7efcc551a640 1 -- 172.21.15.78:0/3677077702 shutdown_connections 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.517+0000 7efcc551a640 1 --2- 172.21.15.78:0/3677077702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc011b140 0x7efcc011d530 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.517+0000 7efcc551a640 1 --2- 172.21.15.78:0/3677077702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc0109830 0x7efcc0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.517+0000 7efcc551a640 1 --2- 172.21.15.78:0/3677077702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.517+0000 7efcc551a640 1 -- 172.21.15.78:0/3677077702 >> 172.21.15.78:0/3677077702 conn(0x7efcc006e140 msgr2=0x7efcc006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.521+0000 7efcc551a640 1 -- 172.21.15.78:0/3677077702 shutdown_connections 2024-09-16T01:24:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.522+0000 7efcc551a640 1 -- 172.21.15.78:0/3677077702 wait complete. 2024-09-16T01:24:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.523+0000 7efcc551a640 1 Processor -- start 2024-09-16T01:24:29.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 -- start start 2024-09-16T01:24:29.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc0109830 0x7efcc01a7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc011b140 0x7efcc01abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efcc0120da0 con 0x7efcc0109830 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efcc0120c20 con 0x7efcc0109450 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.527+0000 7efcc551a640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcc0120f20 con 0x7efcc011b140 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:38096/0 (socket says 172.21.15.78:38096) 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 -- 172.21.15.78:0/1430647641 learned_addr learned my addr 172.21.15.78:0/1430647641 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:29.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc0109830 0x7efcc01a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc011b140 0x7efcc01abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 -- 172.21.15.78:0/1430647641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc011b140 msgr2=0x7efcc01abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc011b140 0x7efcc01abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 -- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc0109830 msgr2=0x7efcc01a7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc0109830 0x7efcc01a7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.528+0000 7efcbeffd640 1 -- 172.21.15.78:0/1430647641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc01ac270 con 0x7efcc0109450 2024-09-16T01:24:29.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.532+0000 7efcbeffd640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efcb400e9f0 tx=0x7efcb400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.534+0000 7efca7fff640 1 -- 172.21.15.78:0/1430647641 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb400cd80 con 0x7efcc0109450 2024-09-16T01:24:29.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.534+0000 7efca7fff640 1 -- 172.21.15.78:0/1430647641 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb4010910 con 0x7efcc0109450 2024-09-16T01:24:29.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.534+0000 7efca7fff640 1 -- 172.21.15.78:0/1430647641 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4018990 con 0x7efcc0109450 2024-09-16T01:24:29.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.536+0000 7efcc551a640 1 -- 172.21.15.78:0/1430647641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc01ac560 con 0x7efcc0109450 2024-09-16T01:24:29.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.536+0000 7efcc551a640 1 -- 172.21.15.78:0/1430647641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc01b3e40 con 0x7efcc0109450 2024-09-16T01:24:29.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.539+0000 7efcc78f3740 1 Processor -- start 2024-09-16T01:24:29.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.539+0000 7efcc78f3740 1 -- start start 2024-09-16T01:24:29.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f2296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f44f1842a0 con 0x55f44f21f360 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f44f12bb10 con 0x55f44f229c20 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.540+0000 7efcc78f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f44f180260 con 0x55f44f21ea40 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.542+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.542+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:38110/0 (socket says 172.21.15.78:38110) 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.542+0000 7efcbe7fc640 1 -- 172.21.15.78:0/1169920942 learned_addr learned my addr 172.21.15.78:0/1169920942 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.542+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.542+0000 7efcbeffd640 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f2296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 -- 172.21.15.78:0/1169920942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 msgr2=0x55f44f21ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 -- 172.21.15.78:0/1169920942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 msgr2=0x55f44f2296e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f2296e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 -- 172.21.15.78:0/1169920942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f44eef5b90 con 0x55f44f229c20 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.543+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009d90 tx=0x7efcb800f7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f41ee59060a48c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.544+0000 7efca7fff640 1 -- 172.21.15.78:0/1430647641 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7efcb4010ac0 con 0x7efcc0109450 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.544+0000 7efca4ff9640 1 -- 172.21.15.78:0/1169920942 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8004b20 con 0x55f44f229c20 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.544+0000 7efca4ff9640 1 -- 172.21.15.78:0/1169920942 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb8004cc0 con 0x55f44f229c20 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 -- 172.21.15.78:0/1169920942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 msgr2=0x55f44f22c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009d90 tx=0x7efcb800f7d0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 -- 172.21.15.78:0/1169920942 shutdown_connections 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f229c20 0x55f44f22c010 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f2296e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 --2- 172.21.15.78:0/1169920942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.545+0000 7efcc78f3740 1 -- 172.21.15.78:0/1169920942 >> 172.21.15.78:0/1169920942 conn(0x55f44f21afd0 msgr2=0x55f44f21b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.547+0000 7efca7fff640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 0x7efca80a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.547+0000 7efca7fff640 1 -- 172.21.15.78:0/1430647641 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7efcb40d0680 con 0x7efcc0109450 2024-09-16T01:24:29.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.547+0000 7efcc78f3740 1 -- 172.21.15.78:0/1169920942 shutdown_connections 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.548+0000 7efcc78f3740 1 -- 172.21.15.78:0/1169920942 wait complete. 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.548+0000 7efcc78f3740 1 -- start start 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.548+0000 7efcc78f3740 1 Processor -- start 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21ea40 0x55f44f22bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f2294e0 0x55f44f229960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f44f22c760 con 0x55f44f21ea40 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f44f10c6e0 con 0x55f44f21f360 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f44f22c8e0 con 0x55f44f2294e0 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcc78f3740 10 client.0 _ll_register_callbacks cb 0x55f44f130a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:38116/0 (socket says 172.21.15.78:38116) 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.549+0000 7efcbe7fc640 1 -- 172.21.15.78:0/3317003622 learned_addr learned my addr 172.21.15.78:0/3317003622 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f2294e0 0x55f44f229960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 -- 172.21.15.78:0/3317003622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f2294e0 msgr2=0x55f44f229960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f2294e0 0x55f44f229960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 -- 172.21.15.78:0/3317003622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21ea40 msgr2=0x55f44f22bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21ea40 0x55f44f22bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 -- 172.21.15.78:0/3317003622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f44f235c20 con 0x55f44f21f360 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009b00 tx=0x7efcb80049a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351d06c634dc42a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.550+0000 7efca2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.551+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8025020 con 0x55f44f21f360 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.551+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f44f1919d0 con 0x55f44f21f360 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.551+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7efcb8004d50 con 0x55f44f21f360 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.551+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efcb8030050 con 0x55f44f21f360 2024-09-16T01:24:29.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.551+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 0x7efca80a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.552+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 0x7efca80a5940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efcb8026000 tx=0x7efcb8022000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.553+0000 7efcc78f3740 10 client.26195 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:29.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.553+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f44f238600 con 0x55f44f21f360 2024-09-16T01:24:29.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.556+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f44f1919d0 con 0x55f44f21f360 2024-09-16T01:24:29.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.556+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efcb800b040 con 0x55f44f21f360 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 10 client.26195 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 10 client.26195 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 20 client.26195 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f239a10 0x55f44f23be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 4 client.26195 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f44f18bd20 con 0x55f44f239a10 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.557+0000 7efcbeffd640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f239a10 0x55f44f23be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.558+0000 7efcbeffd640 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f239a10 0x55f44f23be00 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54ae6226b6a3b1bc server_cookie=34bc44159543681b in_seq=0 out_seq=0 2024-09-16T01:24:29.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.558+0000 7efca2ff5640 10 client.26195 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efca2ff5640 1 -- 172.21.15.78:0/3317003622 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f44f18bd20 con 0x55f44f239a10 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efca2ff5640 10 client.26195 handle_command_reply: tid=0 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 client.26195 shutdown 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 msgr2=0x55f44f228fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009b00 tx=0x7efcb80049a0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 shutdown_connections 2024-09-16T01:24:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f239a10 0x55f44f23be00 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f2294e0 0x55f44f229960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21f360 0x55f44f228fa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3317003622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21ea40 0x55f44f22bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 >> 172.21.15.78:0/3317003622 conn(0x55f44f21afd0 msgr2=0x55f44f21a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efca2ff5640 0 client.26195 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 shutdown_connections 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.559+0000 7efcc78f3740 1 -- 172.21.15.78:0/3317003622 wait complete. 2024-09-16T01:24:29.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.560+0000 7efcc78f3740 20 client.26195 trim_cache size 0 max 16384 2024-09-16T01:24:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.595+0000 7efcc78f3740 1 Processor -- start 2024-09-16T01:24:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.595+0000 7efcc78f3740 1 -- start start 2024-09-16T01:24:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.595+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.595+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f2294e0 0x55f44f2415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:43704/0 (socket says 172.21.15.78:43704) 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58816/0 (socket says 172.21.15.78:58816) 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcc78f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01b8110 con 0x55f44f21f360 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcc78f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01b59e0 con 0x55f44f2294e0 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcc78f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcc0118970 con 0x55f44f21ea40 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbeffd640 1 -- 172.21.15.78:0/931400285 learned_addr learned my addr 172.21.15.78:0/931400285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f2294e0 0x55f44f2415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 -- 172.21.15.78:0/931400285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 msgr2=0x55f44f21b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 -- 172.21.15.78:0/931400285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f2294e0 msgr2=0x55f44f2415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f2294e0 0x55f44f2415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.596+0000 7efcbe7fc640 1 -- 172.21.15.78:0/931400285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f44f228f40 con 0x55f44f21f360 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.597+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55f44f22a550 tx=0x7efcb800dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c933456d67219a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.597+0000 7efca47f8640 1 -- 172.21.15.78:0/931400285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8013070 con 0x55f44f21f360 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.597+0000 7efca47f8640 1 -- 172.21.15.78:0/931400285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb800fcd0 con 0x55f44f21f360 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.597+0000 7efca47f8640 1 -- 172.21.15.78:0/931400285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8025050 con 0x55f44f21f360 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.598+0000 7efcc78f3740 1 -- 172.21.15.78:0/931400285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 msgr2=0x55f44f21a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.598+0000 7efcc78f3740 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55f44f22a550 tx=0x7efcb800dec0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.600+0000 7efcc78f3740 1 -- 172.21.15.78:0/931400285 shutdown_connections 2024-09-16T01:24:29.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.600+0000 7efcc78f3740 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f2294e0 0x55f44f2415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.600+0000 7efcc78f3740 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f21f360 0x55f44f21a200 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.600+0000 7efcc78f3740 1 --2- 172.21.15.78:0/931400285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21ea40 0x55f44f21b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.600+0000 7efcc78f3740 1 -- 172.21.15.78:0/931400285 >> 172.21.15.78:0/931400285 conn(0x55f44f21afd0 msgr2=0x55f44f219af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.604+0000 7efcc78f3740 1 -- 172.21.15.78:0/931400285 shutdown_connections 2024-09-16T01:24:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.604+0000 7efcc78f3740 1 -- 172.21.15.78:0/931400285 wait complete. 2024-09-16T01:24:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.604+0000 7efcc78f3740 1 -- start start 2024-09-16T01:24:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.604+0000 7efcc78f3740 1 Processor -- start 2024-09-16T01:24:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21ea40 0x55f44f232010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21f360 0x55f44f22d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01b4010 con 0x55f44f2294e0 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01b5310 con 0x55f44f21ea40 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01b4570 con 0x55f44f21f360 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.605+0000 7efcc78f3740 10 client.0 _ll_register_callbacks cb 0x7efcb8009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.609+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.609+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58832/0 (socket says 172.21.15.78:58832) 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.609+0000 7efcbf7fe640 1 -- 172.21.15.78:0/3472699225 learned_addr learned my addr 172.21.15.78:0/3472699225 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.609+0000 7efcbe7fc640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21f360 0x55f44f22d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.612+0000 7efcbeffd640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21ea40 0x55f44f232010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.617+0000 7efcbf7fe640 1 -- 172.21.15.78:0/3472699225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21f360 msgr2=0x55f44f22d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.617+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21f360 0x55f44f22d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.617+0000 7efcbf7fe640 1 -- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21ea40 msgr2=0x55f44f232010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.617+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21ea40 0x55f44f232010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.617+0000 7efcbf7fe640 1 -- 172.21.15.78:0/3472699225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f44f22c060 con 0x55f44f2294e0 2024-09-16T01:24:29.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.621+0000 7efcbf7fe640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009860 tx=0x7efcb00095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf90d458fc89be6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.621+0000 7efca27f4640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.622+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0037040 con 0x55f44f2294e0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.622+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7efcb00305b0 con 0x55f44f2294e0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.622+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb800fcd0 con 0x55f44f2294e0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.624+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efcb00307b0 con 0x55f44f2294e0 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.625+0000 7efcc78f3740 10 client.16920 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:29.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.625+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f44f1f5670 con 0x55f44f2294e0 2024-09-16T01:24:29.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.629+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb0030950 con 0x55f44f2294e0 2024-09-16T01:24:29.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.629+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efcb004e020 con 0x55f44f2294e0 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 10 client.16920 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 10 client.16920 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 20 client.16920 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f2a46d0 0x55f44f2a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 4 client.16920 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}] 2024-09-16T01:24:29.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.632+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}) -- 0x7efcb800fcd0 con 0x55f44f2a46d0 2024-09-16T01:24:29.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.633+0000 7efcbeffd640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f2a46d0 0x55f44f2a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:29.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.636+0000 7efca27f4640 10 client.16920 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:29.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.636+0000 7efcbeffd640 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f2a46d0 0x55f44f2a6af0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=354486a976bb35a server_cookie=f543347bc94c6d8f in_seq=0 out_seq=0 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efca27f4640 1 -- 172.21.15.78:0/3472699225 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7efcb800fcd0 con 0x55f44f2a46d0 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efca27f4640 10 client.16920 handle_command_reply: tid=0 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 client.16920 shutdown 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 msgr2=0x55f44f22d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009860 tx=0x7efcb00095a0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 shutdown_connections 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55f44f2a46d0 0x55f44f2a6af0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f44f2294e0 0x55f44f22d880 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f44f21f360 0x55f44f22d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efca27f4640 0 client.16920 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 --2- 172.21.15.78:0/3472699225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f44f21ea40 0x55f44f232010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.970+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 >> 172.21.15.78:0/3472699225 conn(0x55f44f21afd0 msgr2=0x55f44f21a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 shutdown_connections 2024-09-16T01:24:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efcc78f3740 1 -- 172.21.15.78:0/3472699225 wait complete. 2024-09-16T01:24:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efcc78f3740 20 client.16920 trim_cache size 0 max 16384 2024-09-16T01:24:29.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 50, 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "201556dc": { 2024-09-16T01:24:29.975 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 50, 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:24:29.976 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 146.5, 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:29.977 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:29.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:29.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:29.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:29.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:29.978 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:24:29.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 msgr2=0x7efca80a5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 0x7efca80a5940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efcb8026000 tx=0x7efcb8022000 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 msgr2=0x7efcc01a72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efcb400e9f0 tx=0x7efcb400eeb0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 shutdown_connections 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efca80a3480 0x7efca80a5940 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcc011b140 0x7efcc01abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcc0109830 0x7efcc01a7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 --2- 172.21.15.78:0/1430647641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcc0109450 0x7efcc01a72c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 >> 172.21.15.78:0/1430647641 conn(0x7efcc006e140 msgr2=0x7efcc011be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 shutdown_connections 2024-09-16T01:24:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:29.971+0000 7efca5ffb640 1 -- 172.21.15.78:0/1430647641 wait complete. 2024-09-16T01:24:30.673 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '201556dc', quiesce took 7.0 seconds. Will release after: 19.5 2024-09-16T01:24:30.937 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 795.0 s, 395.5 tps, lat 204.532 ms stddev 252.979 2024-09-16T01:24:31.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:30 smithi078 ceph-mon[30937]: pgmap v431: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-09-16T01:24:31.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:30 smithi148 ceph-mon[35030]: pgmap v431: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-09-16T01:24:31.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:30 smithi193 ceph-mon[35701]: pgmap v431: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-09-16T01:24:33.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:32 smithi148 ceph-mon[35030]: pgmap v432: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-16T01:24:33.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:32 smithi078 ceph-mon[30937]: pgmap v432: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-16T01:24:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:32 smithi193 ceph-mon[35701]: pgmap v432: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.9 MiB/s wr, 512 op/s 2024-09-16T01:24:35.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:34 smithi148 ceph-mon[35030]: pgmap v433: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 4.2 MiB/s wr, 429 op/s 2024-09-16T01:24:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:34 smithi078 ceph-mon[30937]: pgmap v433: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 4.2 MiB/s wr, 429 op/s 2024-09-16T01:24:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:34 smithi193 ceph-mon[35701]: pgmap v433: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 4.2 MiB/s wr, 429 op/s 2024-09-16T01:24:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 800.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 -- 172.21.15.78:0/3717330667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 msgr2=0x7f5a1410e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/3717330667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a1410e990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009cd0 tx=0x7f5a0c02a8e0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 -- 172.21.15.78:0/3717330667 shutdown_connections 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/3717330667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a1410e990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/3717330667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 0x7f5a14101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/3717330667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a14107eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 -- 172.21.15.78:0/3717330667 >> 172.21.15.78:0/3717330667 conn(0x7f5a140fd7e0 msgr2=0x7f5a140ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 -- 172.21.15.78:0/3717330667 shutdown_connections 2024-09-16T01:24:36.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 -- 172.21.15.78:0/3717330667 wait complete. 2024-09-16T01:24:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.027+0000 7f5a1a98d640 1 Processor -- start 2024-09-16T01:24:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 -- start start 2024-09-16T01:24:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 0x7f5a14198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a1419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14111f10 con 0x7f5a14102460 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14111d90 con 0x7f5a14107ad0 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a1a98d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5a14112090 con 0x7f5a14101ac0 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39828/0 (socket says 172.21.15.78:39828) 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 -- 172.21.15.78:0/1643980908 learned_addr learned my addr 172.21.15.78:0/1643980908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a1419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a13fff640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 0x7f5a14198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 -- 172.21.15.78:0/1643980908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 msgr2=0x7f5a14198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 0x7f5a14198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 -- 172.21.15.78:0/1643980908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 msgr2=0x7f5a1419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a1419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a137fe640 1 -- 172.21.15.78:0/1643980908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1419d2f0 con 0x7f5a14102460 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.028+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a1419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800d950 tx=0x7f5a0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a117fa640 1 -- 172.21.15.78:0/1643980908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08017070 con 0x7f5a14102460 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1419d5e0 con 0x7f5a14102460 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a117fa640 1 -- 172.21.15.78:0/1643980908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a08009c30 con 0x7f5a14102460 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a117fa640 1 -- 172.21.15.78:0/1643980908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0800f6b0 con 0x7f5a14102460 2024-09-16T01:24:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.029+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a141a4e70 con 0x7f5a14102460 2024-09-16T01:24:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.031+0000 7f5a117fa640 1 -- 172.21.15.78:0/1643980908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f5a080109d0 con 0x7f5a14102460 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.031+0000 7f5a1d806740 1 Processor -- start 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 -- start start 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5c410 0x55a473a66770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a66cb0 0x55a473a690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a4739c02f0 con 0x55a473a5baf0 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a473968270 con 0x55a473a66cb0 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a1d806740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a4739bc2b0 con 0x55a473a5c410 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a117fa640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 0x7f59fc0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a66cb0 0x55a473a690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39832/0 (socket says 172.21.15.78:39832) 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 -- 172.21.15.78:0/3746751713 learned_addr learned my addr 172.21.15.78:0/3746751713 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a13fff640 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5c410 0x55a473a66770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 0x7f59fc0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a117fa640 1 -- 172.21.15.78:0/1643980908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f5a080cfdd0 con 0x7f5a14102460 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 -- 172.21.15.78:0/3746751713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5c410 msgr2=0x55a473a66770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5c410 0x55a473a66770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 -- 172.21.15.78:0/3746751713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a66cb0 msgr2=0x55a473a690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a66cb0 0x55a473a690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.032+0000 7f5a18f2a640 1 -- 172.21.15.78:0/3746751713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a473967340 con 0x55a473a5baf0 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c004af0 tx=0x7f5a0c031ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5670d1a937e7b6c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.042 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 0x7f59fc0a5920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009a60 tx=0x7f5a0c048000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f59fa7fc640 1 -- 172.21.15.78:0/3746751713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c04e070 con 0x55a473a5baf0 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f59fa7fc640 1 -- 172.21.15.78:0/3746751713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c036040 con 0x55a473a5baf0 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f59fa7fc640 1 -- 172.21.15.78:0/3746751713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c0305a0 con 0x55a473a5baf0 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f5a1d806740 1 -- 172.21.15.78:0/3746751713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 msgr2=0x55a473a5bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.033+0000 7f5a1d806740 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c004af0 tx=0x7f5a0c031ba0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- 172.21.15.78:0/3746751713 shutdown_connections 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a66cb0 0x55a473a690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.043 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5c410 0x55a473a66770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- 172.21.15.78:0/3746751713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a5bed0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- 172.21.15.78:0/3746751713 >> 172.21.15.78:0/3746751713 conn(0x55a473a58060 msgr2=0x55a473a58440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- 172.21.15.78:0/3746751713 shutdown_connections 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- 172.21.15.78:0/3746751713 wait complete. 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- start start 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 Processor -- start 2024-09-16T01:24:36.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a640e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a64620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a473a69880 con 0x55a473a5baf0 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a473949290 con 0x55a473a5c410 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.034+0000 7f5a1d806740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a473a69a00 con 0x55a473a66cb0 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a1d806740 10 client.0 _ll_register_callbacks cb 0x55a47397a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39842/0 (socket says 172.21.15.78:39842) 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 -- 172.21.15.78:0/2092720927 learned_addr learned my addr 172.21.15.78:0/2092720927 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a137fe640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a64620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 -- 172.21.15.78:0/2092720927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 msgr2=0x55a473a64620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a64620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 -- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 msgr2=0x55a473a640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a640e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 -- 172.21.15.78:0/2092720927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4736d3210 con 0x55a473a5baf0 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a13fff640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009d80 tx=0x7f5a040117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435b3705a58703c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f59f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04005220 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f5a040053c0 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a0c036040 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a040055c0 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a1d806740 10 client.16932 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.035+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a47379f620 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c036040 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5a0400f040 con 0x55a473a5baf0 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 10 client.16932 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 10 client.16932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 20 client.16932 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473a76cb0 0x55a473a790e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 4 client.16932 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4739cda20 con 0x55a473a76cb0 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.036+0000 7f5a137fe640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473a76cb0 0x55a473a790e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.037+0000 7f5a137fe640 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473a76cb0 0x55a473a790e0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b165bf98589091f server_cookie=c1db381b3717603 in_seq=0 out_seq=0 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.037+0000 7f59f37fe640 10 client.16932 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.037+0000 7f59f37fe640 1 -- 172.21.15.78:0/2092720927 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4739cda20 con 0x55a473a76cb0 2024-09-16T01:24:36.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.037+0000 7f59f37fe640 10 client.16932 handle_command_reply: tid=0 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.037+0000 7f5a1d806740 1 client.16932 shutdown 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 msgr2=0x55a473a63ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009d80 tx=0x7f5a040117b0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 shutdown_connections 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473a76cb0 0x55a473a790e0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a64620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a640e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f59f37fe640 0 client.16932 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2092720927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5baf0 0x55a473a63ba0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 >> 172.21.15.78:0/2092720927 conn(0x55a473a58060 msgr2=0x55a473a571c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 shutdown_connections 2024-09-16T01:24:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 1 -- 172.21.15.78:0/2092720927 wait complete. 2024-09-16T01:24:36.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.038+0000 7f5a1d806740 20 client.16932 trim_cache size 0 max 16384 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 Processor -- start 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 -- start start 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 0x55a473a5e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a64450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141b1910 con 0x55a473a66cb0 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141af1e0 con 0x55a473a5c410 2024-09-16T01:24:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.052+0000 7f5a1d806740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141ad050 con 0x55a473a5baf0 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39852/0 (socket says 172.21.15.78:39852) 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 -- 172.21.15.78:0/2240486106 learned_addr learned my addr 172.21.15.78:0/2240486106 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a137fe640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a64450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a13fff640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 0x55a473a5e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 -- 172.21.15.78:0/2240486106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 msgr2=0x55a473a5e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 0x55a473a5e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 -- 172.21.15.78:0/2240486106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 msgr2=0x55a473a64450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a64450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 -- 172.21.15.78:0/2240486106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a473a59110 con 0x55a473a66cb0 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a13fff640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 0x55a473a5e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c005f80 tx=0x7f5a0c0045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c1c427c9669336 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f59f9ffb640 1 -- 172.21.15.78:0/2240486106 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c04e040 con 0x55a473a66cb0 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f59f9ffb640 1 -- 172.21.15.78:0/2240486106 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c036050 con 0x55a473a66cb0 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.053+0000 7f59f9ffb640 1 -- 172.21.15.78:0/2240486106 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c030800 con 0x55a473a66cb0 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- 172.21.15.78:0/2240486106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 msgr2=0x55a473a7ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c005f80 tx=0x7f5a0c0045c0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- 172.21.15.78:0/2240486106 shutdown_connections 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a66cb0 0x55a473a7ea40 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5c410 0x55a473a64450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 --2- 172.21.15.78:0/2240486106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a5baf0 0x55a473a5e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- 172.21.15.78:0/2240486106 >> 172.21.15.78:0/2240486106 conn(0x55a473a58060 msgr2=0x55a473a65c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- 172.21.15.78:0/2240486106 shutdown_connections 2024-09-16T01:24:36.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- 172.21.15.78:0/2240486106 wait complete. 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 -- start start 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.054+0000 7f5a1d806740 1 Processor -- start 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5baf0 0x55a473a6f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a6aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141ad6a0 con 0x55a473a5c410 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141aeb10 con 0x55a473a5baf0 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5a141add70 con 0x55a473a66cb0 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 10 client.0 _ll_register_callbacks cb 0x7f5a04009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39862/0 (socket says 172.21.15.78:39862) 2024-09-16T01:24:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 -- 172.21.15.78:0/1569401248 learned_addr learned my addr 172.21.15.78:0/1569401248 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a13fff640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5baf0 0x55a473a6f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a6aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 -- 172.21.15.78:0/1569401248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 msgr2=0x55a473a6aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a6aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 -- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5baf0 msgr2=0x55a473a6f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5baf0 0x55a473a6f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 -- 172.21.15.78:0/1569401248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a473a6b1e0 con 0x55a473a5c410 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a18f2a640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a6aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a137fe640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5a080cd1e0 tx=0x7f5a08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8e36a051135fa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f59f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08020040 con 0x55a473a5c410 2024-09-16T01:24:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f5a080cd3b0 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.055+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a0c036050 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a080b9050 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 10 client.16938 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a473ae1830 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0c036050 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5a0809ab80 con 0x55a473a5c410 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 10 client.16938 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 10 client.16938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 20 client.16938 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473ae1f10 0x55a473ae4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 4 client.16938 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:24:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a4739c7d70 con 0x55a473ae1f10 2024-09-16T01:24:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.056+0000 7f5a13fff640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473ae1f10 0x55a473ae4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.057+0000 7f5a13fff640 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473ae1f10 0x55a473ae4330 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8c3ffb3149f726 server_cookie=54de2aa3b80b9d0f in_seq=0 out_seq=0 2024-09-16T01:24:36.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.057+0000 7f59f37fe640 10 client.16938 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f59f37fe640 1 -- 172.21.15.78:0/1569401248 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a4739c7d70 con 0x55a473ae1f10 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f59f37fe640 10 client.16938 handle_command_reply: tid=0 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 client.16938 shutdown 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 msgr2=0x55a473a6a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5a080cd1e0 tx=0x7f5a08005880 comp rx=0 tx=0).stop 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 shutdown_connections 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a473ae1f10 0x55a473ae4330 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a473a66cb0 0x55a473a6aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f59f37fe640 0 client.16938 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a473a5c410 0x55a473a6a5b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 --2- 172.21.15.78:0/1569401248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a473a5baf0 0x55a473a6f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 >> 172.21.15.78:0/1569401248 conn(0x55a473a58060 msgr2=0x55a473a569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 shutdown_connections 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.065+0000 7f5a1d806740 1 -- 172.21.15.78:0/1569401248 wait complete. 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1d806740 20 client.16938 trim_cache size 0 max 16384 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:24:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:24:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:24:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 msgr2=0x7f59fc0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 0x7f59fc0a5920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009a60 tx=0x7f5a0c048000 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 msgr2=0x7f5a14198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800d950 tx=0x7f5a0800de10 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 shutdown_connections 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f59fc0a3460 0x7f59fc0a5920 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a14107ad0 0x7f5a1419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a14102460 0x7f5a14198840 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 --2- 172.21.15.78:0/1643980908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a14101ac0 0x7f5a14198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 >> 172.21.15.78:0/1643980908 conn(0x7f5a140fd7e0 msgr2=0x7f5a140fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 shutdown_connections 2024-09-16T01:24:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:36.066+0000 7f5a1a98d640 1 -- 172.21.15.78:0/1643980908 wait complete. 2024-09-16T01:24:37.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:36 smithi148 ceph-mon[35030]: pgmap v434: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 3.5 MiB/s wr, 365 op/s 2024-09-16T01:24:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:36 smithi078 ceph-mon[30937]: pgmap v434: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 3.5 MiB/s wr, 365 op/s 2024-09-16T01:24:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:36 smithi193 ceph-mon[35701]: pgmap v434: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 3.5 MiB/s wr, 365 op/s 2024-09-16T01:24:39.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:38 smithi148 ceph-mon[35030]: pgmap v435: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-09-16T01:24:39.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:38 smithi078 ceph-mon[30937]: pgmap v435: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-09-16T01:24:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:38 smithi193 ceph-mon[35701]: pgmap v435: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-09-16T01:24:40.941 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 805.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:24:41.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:40 smithi078 ceph-mon[30937]: pgmap v436: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-16T01:24:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:40 smithi193 ceph-mon[35701]: pgmap v436: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-16T01:24:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:40 smithi148 ceph-mon[35030]: pgmap v436: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 2.2 MiB/s wr, 210 op/s 2024-09-16T01:24:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:42 smithi078 ceph-mon[30937]: pgmap v437: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-09-16T01:24:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:42 smithi193 ceph-mon[35701]: pgmap v437: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-09-16T01:24:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:42 smithi148 ceph-mon[35030]: pgmap v437: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-09-16T01:24:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:44 smithi078 ceph-mon[30937]: pgmap v438: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 468 KiB/s wr, 42 op/s 2024-09-16T01:24:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:44 smithi193 ceph-mon[35701]: pgmap v438: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 468 KiB/s wr, 42 op/s 2024-09-16T01:24:45.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:44 smithi148 ceph-mon[35030]: pgmap v438: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 468 KiB/s wr, 42 op/s 2024-09-16T01:24:45.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 810.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:24:47.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:46 smithi078 ceph-mon[30937]: pgmap v439: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-09-16T01:24:47.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:46 smithi193 ceph-mon[35701]: pgmap v439: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-09-16T01:24:47.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:46 smithi148 ceph-mon[35030]: pgmap v439: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 43 KiB/s wr, 13 op/s 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 -- 172.21.15.78:0/3999242754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b81009f0 msgr2=0x7f16b8100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 --2- 172.21.15.78:0/3999242754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b81009f0 0x7f16b8100e50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009930 tx=0x7f16a002a420 comp rx=0 tx=0).stop 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 -- 172.21.15.78:0/3999242754 shutdown_connections 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 --2- 172.21.15.78:0/3999242754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b8101390 0x7f16b810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 --2- 172.21.15.78:0/3999242754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b81009f0 0x7f16b8100e50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 --2- 172.21.15.78:0/3999242754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8106a00 0x7f16b8106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 -- 172.21.15.78:0/3999242754 >> 172.21.15.78:0/3999242754 conn(0x7f16b80fc6b0 msgr2=0x7f16b80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 -- 172.21.15.78:0/3999242754 shutdown_connections 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 -- 172.21.15.78:0/3999242754 wait complete. 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.160+0000 7f16be9c4640 1 Processor -- start 2024-09-16T01:24:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 -- start start 2024-09-16T01:24:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 0x7f16b81991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8101390 0x7f16b8199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8110e40 con 0x7f16b8106a00 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8110cc0 con 0x7f16b81009f0 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16be9c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8110fc0 con 0x7f16b8101390 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42538/0 (socket says 172.21.15.78:42538) 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 -- 172.21.15.78:0/1299352471 learned_addr learned my addr 172.21.15.78:0/1299352471 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16b77fe640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8101390 0x7f16b8199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16b7fff640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 0x7f16b81991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 -- 172.21.15.78:0/1299352471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8101390 msgr2=0x7f16b8199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8101390 0x7f16b8199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 -- 172.21.15.78:0/1299352471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 msgr2=0x7f16b81991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 0x7f16b81991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16bcf61640 1 -- 172.21.15.78:0/1299352471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b81a4790 con 0x7f16b8106a00 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.161+0000 7f16b7fff640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 0x7f16b81991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16bcf61640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f16b000cce0 tx=0x7f16b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16b57fa640 1 -- 172.21.15.78:0/1299352471 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b000f450 con 0x7f16b8106a00 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16b57fa640 1 -- 172.21.15.78:0/1299352471 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b000fad0 con 0x7f16b8106a00 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16b57fa640 1 -- 172.21.15.78:0/1299352471 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0018c10 con 0x7f16b8106a00 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b81a4960 con 0x7f16b8106a00 2024-09-16T01:24:49.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.162+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b81a4e30 con 0x7f16b8106a00 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.164+0000 7f16b57fa640 1 -- 172.21.15.78:0/1299352471 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f16b000f5f0 con 0x7f16b8106a00 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.164+0000 7f16c183f740 1 Processor -- start 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.164+0000 7f16c183f740 1 -- start start 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf59bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf5002f0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf4a8270 con 0x55b7cf59c410 2024-09-16T01:24:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16c183f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf4fc2b0 con 0x55b7cf59baf0 2024-09-16T01:24:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42548/0 (socket says 172.21.15.78:42548) 2024-09-16T01:24:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 -- 172.21.15.78:0/2419588870 learned_addr learned my addr 172.21.15.78:0/2419588870 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:49.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b57fa640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 0x7f16a40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf59bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b7fff640 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16bcf61640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 0x7f16a40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b57fa640 1 -- 172.21.15.78:0/1299352471 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f16b00d0860 con 0x7f16b8106a00 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 -- 172.21.15.78:0/2419588870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 msgr2=0x55b7cf59bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf59bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 -- 172.21.15.78:0/2419588870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 msgr2=0x55b7cf5a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.165+0000 7f16b77fe640 1 -- 172.21.15.78:0/2419588870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7cf4a7340 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f16a002a940 tx=0x7f16a0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10094a00638df7bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16bcf61640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 0x7f16a40a5920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16b009c3f0 tx=0x7f16b006e470 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16ae7fc640 1 -- 172.21.15.78:0/2419588870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0035070 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16ae7fc640 1 -- 172.21.15.78:0/2419588870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0031730 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16ae7fc640 1 -- 172.21.15.78:0/2419588870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00047a0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 -- 172.21.15.78:0/2419588870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 msgr2=0x55b7cf5a90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f16a002a940 tx=0x7f16a0005c00 comp rx=0 tx=0).stop 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 -- 172.21.15.78:0/2419588870 shutdown_connections 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a90a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 --2- 172.21.15.78:0/2419588870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf59bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.166+0000 7f16c183f740 1 -- 172.21.15.78:0/2419588870 >> 172.21.15.78:0/2419588870 conn(0x55b7cf598060 msgr2=0x55b7cf598440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- 172.21.15.78:0/2419588870 shutdown_connections 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- 172.21.15.78:0/2419588870 wait complete. 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- start start 2024-09-16T01:24:49.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 Processor -- start 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf5a9880 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf489290 con 0x55b7cf59c410 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf5a9a00 con 0x55b7cf59baf0 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16c183f740 10 client.0 _ll_register_callbacks cb 0x55b7cf4ba140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.167+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42560/0 (socket says 172.21.15.78:42560) 2024-09-16T01:24:49.178 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 -- 172.21.15.78:0/252097094 learned_addr learned my addr 172.21.15.78:0/252097094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b7fff640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 -- 172.21.15.78:0/252097094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 msgr2=0x55b7cf5a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 -- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 msgr2=0x55b7cf5a4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 -- 172.21.15.78:0/252097094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7cf213210 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16b77fe640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f16a0006010 tx=0x7f16a0005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d099658ecb52eecc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16977fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0035070 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f16a0004250 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a0031730 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f16a005b050 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16c183f740 10 client.16950 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b7cf2df620 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.168+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0031730 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f16a003d4e0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 10 client.16950 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 10 client.16950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 20 client.16950 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf5b6b00 0x55b7cf5b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 4 client.16950 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7cf50da20 con 0x55b7cf5b6b00 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16b7fff640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf5b6b00 0x55b7cf5b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16977fe640 10 client.16950 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.169+0000 7f16b7fff640 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf5b6b00 0x55b7cf5b8ef0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b8f817866bc7441 server_cookie=31c416ecba6a9ddb in_seq=0 out_seq=0 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16977fe640 1 -- 172.21.15.78:0/252097094 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b7cf50da20 con 0x55b7cf5b6b00 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16977fe640 10 client.16950 handle_command_reply: tid=0 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16c183f740 1 client.16950 shutdown 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 msgr2=0x55b7cf5a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f16a0006010 tx=0x7f16a0005360 comp rx=0 tx=0).stop 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 shutdown_connections 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.170+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf5b6b00 0x55b7cf5b8ef0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a4660 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16977fe640 0 client.16950 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 --2- 172.21.15.78:0/252097094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 >> 172.21.15.78:0/252097094 conn(0x55b7cf598060 msgr2=0x55b7cf59dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 shutdown_connections 2024-09-16T01:24:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 1 -- 172.21.15.78:0/252097094 wait complete. 2024-09-16T01:24:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.171+0000 7f16c183f740 20 client.16950 trim_cache size 0 max 16384 2024-09-16T01:24:49.191 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 Processor -- start 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 -- start start 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81b18b0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81af180 con 0x55b7cf59baf0 2024-09-16T01:24:49.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16c183f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81acff0 con 0x55b7cf59c410 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42574/0 (socket says 172.21.15.78:42574) 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.185+0000 7f16b77fe640 1 -- 172.21.15.78:0/2487376564 learned_addr learned my addr 172.21.15.78:0/2487376564 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 -- 172.21.15.78:0/2487376564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59c410 msgr2=0x55b7cf5a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 -- 172.21.15.78:0/2487376564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59baf0 msgr2=0x55b7cf5a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 -- 172.21.15.78:0/2487376564 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7cf597e90 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16b77fe640 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005430 tx=0x7f16a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177c1d93cd03a286 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16adffb640 1 -- 172.21.15.78:0/2487376564 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0035040 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16adffb640 1 -- 172.21.15.78:0/2487376564 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0057020 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.186+0000 7f16adffb640 1 -- 172.21.15.78:0/2487376564 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0031bb0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- 172.21.15.78:0/2487376564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 msgr2=0x55b7cf5a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005430 tx=0x7f16a0000bc0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- 172.21.15.78:0/2487376564 shutdown_connections 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5a0e40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59c410 0x55b7cf5a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- 172.21.15.78:0/2487376564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59baf0 0x55b7cf5a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- 172.21.15.78:0/2487376564 >> 172.21.15.78:0/2487376564 conn(0x55b7cf598060 msgr2=0x55b7cf5b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- 172.21.15.78:0/2487376564 shutdown_connections 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- 172.21.15.78:0/2487376564 wait complete. 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- start start 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 Processor -- start 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5ba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81ad640 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81aeab0 con 0x55b7cf59c410 2024-09-16T01:24:49.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16b81add10 con 0x55b7cf59baf0 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.187+0000 7f16c183f740 10 client.0 _ll_register_callbacks cb 0x7f16a8009320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42586/0 (socket says 172.21.15.78:42586) 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 -- 172.21.15.78:0/3184976336 learned_addr learned my addr 172.21.15.78:0/3184976336 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b7fff640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 -- 172.21.15.78:0/3184976336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 msgr2=0x55b7cf5b9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5b9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 -- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 msgr2=0x55b7cf5ba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5ba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 -- 172.21.15.78:0/3184976336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7cf620690 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16b77fe640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005c40 tx=0x7f16a0032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c456ad911c04927 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16977fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0035070 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f16a0058040 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a0057020 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f16a003d510 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16c183f740 10 client.16956 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.188+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b7cf621980 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0057020 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f16a00035f0 con 0x55b7cf5a6cb0 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 10 client.16956 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 10 client.16956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 20 client.16956 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:49.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf621e50 0x55b7cf624240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 4 client.16956 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:24:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7cf507d70 con 0x55b7cf621e50 2024-09-16T01:24:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.189+0000 7f16b7fff640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf621e50 0x55b7cf624240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.190+0000 7f16977fe640 10 client.16956 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:49.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.190+0000 7f16b7fff640 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf621e50 0x55b7cf624240 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=473d0eadd62a5652 server_cookie=785bd279c74108f8 in_seq=0 out_seq=0 2024-09-16T01:24:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.197+0000 7f16977fe640 1 -- 172.21.15.78:0/3184976336 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7cf507d70 con 0x55b7cf621e50 2024-09-16T01:24:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.197+0000 7f16977fe640 10 client.16956 handle_command_reply: tid=0 2024-09-16T01:24:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.197+0000 7f16c183f740 1 client.16956 shutdown 2024-09-16T01:24:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.197+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 msgr2=0x55b7cf5ba950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.197+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f16a0005c40 tx=0x7f16a0032d70 comp rx=0 tx=0).stop 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 shutdown_connections 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55b7cf621e50 0x55b7cf624240 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7cf5a6cb0 0x55b7cf5ba950 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7cf59c410 0x55b7cf5ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16977fe640 0 client.16956 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 --2- 172.21.15.78:0/3184976336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b7cf59baf0 0x55b7cf5b9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 >> 172.21.15.78:0/3184976336 conn(0x55b7cf598060 msgr2=0x55b7cf5b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 shutdown_connections 2024-09-16T01:24:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 1 -- 172.21.15.78:0/3184976336 wait complete. 2024-09-16T01:24:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16c183f740 20 client.16956 trim_cache size 0 max 16384 2024-09-16T01:24:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:24:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:24:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:24:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:24:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 msgr2=0x7f16a40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 0x7f16a40a5920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16b009c3f0 tx=0x7f16b006e470 comp rx=0 tx=0).stop 2024-09-16T01:24:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 msgr2=0x7f16b81a4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f16b000cce0 tx=0x7f16b000a620 comp rx=0 tx=0).stop 2024-09-16T01:24:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 shutdown_connections 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f16a40a3460 0x7f16a40a5920 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b8106a00 0x7f16b81a4250 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16b8101390 0x7f16b8199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 --2- 172.21.15.78:0/1299352471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16b81009f0 0x7f16b81991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 >> 172.21.15.78:0/1299352471 conn(0x7f16b80fc6b0 msgr2=0x7f16b80fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 shutdown_connections 2024-09-16T01:24:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:24:49.198+0000 7f16be9c4640 1 -- 172.21.15.78:0/1299352471 wait complete. 2024-09-16T01:24:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:48 smithi078 ceph-mon[30937]: pgmap v440: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:49.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:48 smithi193 ceph-mon[35701]: pgmap v440: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:48 smithi148 ceph-mon[35030]: pgmap v440: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:50.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:24:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:24:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 815.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:24:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 -- 172.21.15.78:0/4211147601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0106f10 msgr2=0x7f08b010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 --2- 172.21.15.78:0/4211147601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0106f10 0x7f08b010ea90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009900 tx=0x7f08a802a410 comp rx=0 tx=0).stop 2024-09-16T01:24:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 -- 172.21.15.78:0/4211147601 shutdown_connections 2024-09-16T01:24:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 --2- 172.21.15.78:0/4211147601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0106f10 0x7f08b010ea90 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 --2- 172.21.15.78:0/4211147601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0102790 0x7f08b01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 --2- 172.21.15.78:0/4211147601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 0x7f08b01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 -- 172.21.15.78:0/4211147601 >> 172.21.15.78:0/4211147601 conn(0x7f08b00fd8c0 msgr2=0x7f08b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 -- 172.21.15.78:0/4211147601 shutdown_connections 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.969+0000 7f08b5365640 1 -- 172.21.15.78:0/4211147601 wait complete. 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 Processor -- start 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 -- start start 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 0x7f08b0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 0x7f08b019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0111f90 con 0x7f08b0102790 2024-09-16T01:24:51.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0111e10 con 0x7f08b0101de0 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.970+0000 7f08b5365640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0112110 con 0x7f08b0106f10 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42594/0 (socket says 172.21.15.78:42594) 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 -- 172.21.15.78:0/3286436823 learned_addr learned my addr 172.21.15.78:0/3286436823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08aeffd640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 0x7f08b0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08af7fe640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 0x7f08b019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 -- 172.21.15.78:0/3286436823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 msgr2=0x7f08b019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 0x7f08b019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 -- 172.21.15.78:0/3286436823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 msgr2=0x7f08b0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 0x7f08b0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 -- 172.21.15.78:0/3286436823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b019d420 con 0x7f08b0102790 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08af7fe640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 0x7f08b019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:24:51.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08ae7fc640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f08a400b4a0 tx=0x7f08a400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4014070 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b019d710 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a4010af0 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4017d10 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.971+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b01a4ff0 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.972+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b01021c0 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.973+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f08a4017070 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.974+0000 7f0897fff640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 0x7f08980a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.974+0000 7f08aeffd640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 0x7f08980a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.974+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f08a40cf760 con 0x7f08b0102790 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.975+0000 7f08aeffd640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 0x7f08980a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08a0005c40 tx=0x7f08a000a590 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:51.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:50.977+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f08a409be60 con 0x7f08b0102790 2024-09-16T01:24:51.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.179+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08b01997d0 con 0x7f08b0102790 2024-09-16T01:24:51.180 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:50 smithi078 ceph-mon[30937]: pgmap v441: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:51.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.180+0000 7f0897fff640 1 -- 172.21.15.78:0/3286436823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f08b01997d0 con 0x7f08b0102790 2024-09-16T01:24:51.181 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:24:51.181 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:24:51.182 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 msgr2=0x7f08980a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 0x7f08980a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08a0005c40 tx=0x7f08a000a590 comp rx=0 tx=0).stop 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 msgr2=0x7f08b01989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f08a400b4a0 tx=0x7f08a400b960 comp rx=0 tx=0).stop 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 shutdown_connections 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08980a33b0 0x7f08980a5870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08b0106f10 0x7f08b019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08b0102790 0x7f08b01989b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 --2- 172.21.15.78:0/3286436823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08b0101de0 0x7f08b0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:51.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 >> 172.21.15.78:0/3286436823 conn(0x7f08b00fd8c0 msgr2=0x7f08b00fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:51.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 shutdown_connections 2024-09-16T01:24:51.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:51.183+0000 7f08b5365640 1 -- 172.21.15.78:0/3286436823 wait complete. 2024-09-16T01:24:51.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:50 smithi193 ceph-mon[35701]: pgmap v441: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:51.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:50 smithi148 ceph-mon[35030]: pgmap v441: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.4 KiB/s wr, 8 op/s 2024-09-16T01:24:51.992 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 201556dc --release --await' 2024-09-16T01:24:51.993 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 201556dc --release --await 2024-09-16T01:24:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:24:52.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:51 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3286436823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:52.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:51 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3286436823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:52.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:51 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3286436823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:24:52.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 -- 172.21.15.78:0/2869326325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4102790 msgr2=0x7f64f41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/2869326325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4102790 0x7f64f41067e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009900 tx=0x7f64f002a410 comp rx=0 tx=0).stop 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 -- 172.21.15.78:0/2869326325 shutdown_connections 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/2869326325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4106f10 0x7f64f410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/2869326325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4102790 0x7f64f41067e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/2869326325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4101de0 0x7f64f41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 -- 172.21.15.78:0/2869326325 >> 172.21.15.78:0/2869326325 conn(0x7f64f40fd8c0 msgr2=0x7f64f40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 -- 172.21.15.78:0/2869326325 shutdown_connections 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.726+0000 7f64fb2d2640 1 -- 172.21.15.78:0/2869326325 wait complete. 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 Processor -- start 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 -- start start 2024-09-16T01:24:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 0x7f64f4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4106f10 0x7f64f4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4111f70 con 0x7f64f4101de0 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4111df0 con 0x7f64f4106f10 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64fb2d2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41120f0 con 0x7f64f4102790 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42610/0 (socket says 172.21.15.78:42610) 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 -- 172.21.15.78:0/3878270243 learned_addr learned my addr 172.21.15.78:0/3878270243 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f986f640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4106f10 0x7f64f4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f886d640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 0x7f64f4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 -- 172.21.15.78:0/3878270243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 msgr2=0x7f64f4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 0x7f64f4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 -- 172.21.15.78:0/3878270243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4106f10 msgr2=0x7f64f4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4106f10 0x7f64f4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f906e640 1 -- 172.21.15.78:0/3878270243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f41a69b0 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.727+0000 7f64f886d640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 0x7f64f4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64f906e640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f64e400d950 tx=0x7f64e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64ea7fc640 1 -- 172.21.15.78:0/3878270243 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4017070 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64ea7fc640 1 -- 172.21.15.78:0/3878270243 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e4009c30 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f41a6ca0 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64ea7fc640 1 -- 172.21.15.78:0/3878270243 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e400f6b0 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.728+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f41a71d0 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64ea7fc640 1 -- 172.21.15.78:0/3878270243 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f64e40109d0 con 0x7f64f4101de0 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 Processor -- start 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 -- start start 2024-09-16T01:24:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 0x55e9163fb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e916353620 con 0x55e9163ee060 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9162fae50 con 0x55e9163f9240 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e91634f5e0 con 0x55e9163ee980 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42618/0 (socket says 172.21.15.78:42618) 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 -- 172.21.15.78:0/4025862219 learned_addr learned my addr 172.21.15.78:0/4025862219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f886d640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 0x55e9163fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f906e640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 -- 172.21.15.78:0/4025862219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 msgr2=0x55e9163f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 -- 172.21.15.78:0/4025862219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 msgr2=0x55e9163fb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64ea7fc640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 0x7f64e00a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.731+0000 7f64f986f640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 0x55e9163fb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64f986f640 1 -- 172.21.15.78:0/4025862219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9162ebd40 con 0x55e9163ee060 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64f906e640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64f886d640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 0x55e9163fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64f986f640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 0x7f64e00a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64ea7fc640 1 -- 172.21.15.78:0/3878270243 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f64e40cfd50 con 0x7f64f4101de0 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64f986f640 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009f20 tx=0x7f64ec00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9673d3d6c5521 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64df7fe640 1 -- 172.21.15.78:0/4025862219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec021070 con 0x55e9163ee060 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64df7fe640 1 -- 172.21.15.78:0/4025862219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec0103d0 con 0x55e9163ee060 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.732+0000 7f64df7fe640 1 -- 172.21.15.78:0/4025862219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec015070 con 0x55e9163ee060 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- 172.21.15.78:0/4025862219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 msgr2=0x55e9163ee440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64f986f640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 0x7f64e00a58f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f64ec01e000 tx=0x7f64ec021000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009f20 tx=0x7f64ec00f7f0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- 172.21.15.78:0/4025862219 shutdown_connections 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163f9240 0x55e9163fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/4025862219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163ee440 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- 172.21.15.78:0/4025862219 >> 172.21.15.78:0/4025862219 conn(0x55e9163ea5f0 msgr2=0x55e9163ea9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- 172.21.15.78:0/4025862219 shutdown_connections 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- 172.21.15.78:0/4025862219 wait complete. 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 -- start start 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.733+0000 7f64fd6ab740 1 Processor -- start 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee060 0x55e9163f6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e9163fbe10 con 0x55e9163f9240 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9162dbf10 con 0x55e9163ee060 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e9163fbf90 con 0x55e9163ee980 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 10 client.0 _ll_register_callbacks cb 0x55e9162ffdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42632/0 (socket says 172.21.15.78:42632) 2024-09-16T01:24:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 -- 172.21.15.78:0/3889241416 learned_addr learned my addr 172.21.15.78:0/3889241416 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f906e640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee060 0x55e9163f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f886d640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 -- 172.21.15.78:0/3889241416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 msgr2=0x55e9163f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 -- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee060 msgr2=0x55e9163f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee060 0x55e9163f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 -- 172.21.15.78:0/3889241416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e916062580 con 0x55e9163f9240 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f886d640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64f986f640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00a010 tx=0x7f64ec0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393cd663ad4be2eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec015070 con 0x55e9163f9240 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ec0103d0 con 0x55e9163f9240 2024-09-16T01:24:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f64ec0049e0 con 0x55e9163f9240 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64ec010990 con 0x55e9163f9240 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 10 client.16974 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.734+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e91612c960 con 0x55e9163f9240 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec0103d0 con 0x55e9163f9240 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f64ec02d070 con 0x55e9163f9240 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 10 client.16974 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 10 client.16974 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 20 client.16974 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e916408c20 0x55e91640b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 4 client.16974 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e916360b90 con 0x55e916408c20 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.735+0000 7f64f906e640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e916408c20 0x55e91640b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64dcff9640 10 client.16974 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64f906e640 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e916408c20 0x55e91640b010 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efd4e9d24736167e server_cookie=f932252d3188d967 in_seq=0 out_seq=0 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64dcff9640 1 -- 172.21.15.78:0/3889241416 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e916360b90 con 0x55e916408c20 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64dcff9640 10 client.16974 handle_command_reply: tid=0 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 client.16974 shutdown 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 msgr2=0x55e9163f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00a010 tx=0x7f64ec0119d0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 shutdown_connections 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e916408c20 0x55e91640b010 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e9163f6b90 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee980 0x55e9163f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/3889241416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee060 0x55e9163f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 >> 172.21.15.78:0/3889241416 conn(0x55e9163ea5f0 msgr2=0x55e9163f02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.736+0000 7f64dcff9640 0 client.16974 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.737+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 shutdown_connections 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.737+0000 7f64fd6ab740 1 -- 172.21.15.78:0/3889241416 wait complete. 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.737+0000 7f64fd6ab740 20 client.16974 trim_cache size 0 max 16384 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 Processor -- start 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 -- start start 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee060 0x55e9163f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163e9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41b3dc0 con 0x55e9163f9240 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41b1690 con 0x55e9163ee980 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41af500 con 0x55e9163ee060 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64f986f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42646/0 (socket says 172.21.15.78:42646) 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.753+0000 7f64f986f640 1 -- 172.21.15.78:0/447806117 learned_addr learned my addr 172.21.15.78:0/447806117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f886d640 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f906e640 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee060 0x55e9163f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 -- 172.21.15.78:0/447806117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee060 msgr2=0x55e9163f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee060 0x55e9163f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 -- 172.21.15.78:0/447806117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 msgr2=0x55e9163e9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163e9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 -- 172.21.15.78:0/447806117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9163ee7b0 con 0x55e9163f9240 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64f986f640 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f64ec013000 tx=0x7f64ec00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1a908d4da849fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64deffd640 1 -- 172.21.15.78:0/447806117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec015040 con 0x55e9163f9240 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64deffd640 1 -- 172.21.15.78:0/447806117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec0103d0 con 0x55e9163f9240 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.754+0000 7f64deffd640 1 -- 172.21.15.78:0/447806117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec01b9a0 con 0x55e9163f9240 2024-09-16T01:24:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- 172.21.15.78:0/447806117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 msgr2=0x55e916410780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f64ec013000 tx=0x7f64ec00f7f0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- 172.21.15.78:0/447806117 shutdown_connections 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163f9240 0x55e916410780 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163e9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/447806117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163ee060 0x55e9163f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- 172.21.15.78:0/447806117 >> 172.21.15.78:0/447806117 conn(0x55e9163ea5f0 msgr2=0x55e9163e8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- 172.21.15.78:0/447806117 shutdown_connections 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- 172.21.15.78:0/447806117 wait complete. 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- start start 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 Processor -- start 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163fdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163f9240 0x55e91646c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41afb50 con 0x55e9163ee060 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41b0fc0 con 0x55e9163ee980 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41b0220 con 0x55e9163f9240 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64fd6ab740 10 client.0 _ll_register_callbacks cb 0x7f64ec010f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64f906e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.755+0000 7f64f906e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42654/0 (socket says 172.21.15.78:42654) 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 -- 172.21.15.78:0/2158292380 learned_addr learned my addr 172.21.15.78:0/2158292380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f986f640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163f9240 0x55e91646c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f886d640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163fdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 -- 172.21.15.78:0/2158292380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163f9240 msgr2=0x55e91646c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163f9240 0x55e91646c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 -- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 msgr2=0x55e9163fdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:24:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163fdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 -- 172.21.15.78:0/2158292380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e91646cd10 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64f906e640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009610 tx=0x7f64e400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469a8f65800fb2dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4020040 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f64e40982b0 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ec0103d0 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64e409bc40 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64fd6ab740 10 client.16980 fetch_fsmap learned FSMap version 31 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e9164740d0 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.756+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec0103d0 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f64e409b220 con 0x55e9163ee060 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 10 client.16980 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:24:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 10 client.16980 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 20 client.16980 populate_metadata read hostname 'smithi078' 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e9164747d0 0x55e916476bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 4 client.16980 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "201556dc", "release": true, "await": true}] 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "201556dc", "release": true, "await": true}) -- 0x55e91635b0a0 con 0x55e9164747d0 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64f886d640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e9164747d0 0x55e916476bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64dcff9640 10 client.16980 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.757+0000 7f64f886d640 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e9164747d0 0x55e916476bf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13d0175db61c3126 server_cookie=a03ffe5091a56656 in_seq=0 out_seq=0 2024-09-16T01:24:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.818+0000 7f64dcff9640 1 -- 172.21.15.78:0/2158292380 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e91635b0a0 con 0x55e9164747d0 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.818+0000 7f64dcff9640 10 client.16980 handle_command_reply: tid=0 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.818+0000 7f64fd6ab740 1 client.16980 shutdown 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.818+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 msgr2=0x55e9163fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.818+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009610 tx=0x7f64e400bf60 comp rx=0 tx=0).stop 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 shutdown_connections 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55e9164747d0 0x55e916476bf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e9163f9240 0x55e91646c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9163ee980 0x55e9163fdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 --2- 172.21.15.78:0/2158292380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e9163ee060 0x55e9163fd7c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64dcff9640 0 client.16980 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:24:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.819+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 >> 172.21.15.78:0/2158292380 conn(0x55e9163ea5f0 msgr2=0x55e9163f68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.822+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 shutdown_connections 2024-09-16T01:24:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.822+0000 7f64fd6ab740 1 -- 172.21.15.78:0/2158292380 wait complete. 2024-09-16T01:24:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.822+0000 7f64fd6ab740 20 client.16980 trim_cache size 0 max 16384 2024-09-16T01:24:52.823 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 52, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "201556dc": { 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 52, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:24:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 146.5, 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:24:52.825 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:24:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 msgr2=0x7f64e00a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 0x7f64e00a58f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f64ec01e000 tx=0x7f64ec021000 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 msgr2=0x7f64f40744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f64e400d950 tx=0x7f64e400de10 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 shutdown_connections 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f64e00a3430 0x7f64e00a58f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64f4106f10 0x7f64f4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64f4102790 0x7f64f4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 --2- 172.21.15.78:0/3878270243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64f4101de0 0x7f64f40744f0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 >> 172.21.15.78:0/3878270243 conn(0x7f64f40fd8c0 msgr2=0x7f64f40fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 shutdown_connections 2024-09-16T01:24:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:24:52.823+0000 7f64fb2d2640 1 -- 172.21.15.78:0/3878270243 wait complete. 2024-09-16T01:24:53.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:52 smithi078 ceph-mon[30937]: pgmap v442: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2024-09-16T01:24:53.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:52 smithi193 ceph-mon[35701]: pgmap v442: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2024-09-16T01:24:53.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:52 smithi148 ceph-mon[35030]: pgmap v442: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2024-09-16T01:24:53.616 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '201556dc', total seconds elapsed: 29.9 2024-09-16T01:24:55.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:54 smithi078 ceph-mon[30937]: pgmap v443: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 39 KiB/s wr, 12 op/s 2024-09-16T01:24:55.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:54 smithi193 ceph-mon[35701]: pgmap v443: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 39 KiB/s wr, 12 op/s 2024-09-16T01:24:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:54 smithi148 ceph-mon[35030]: pgmap v443: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 39 KiB/s wr, 12 op/s 2024-09-16T01:24:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 820.0 s, 434.0 tps, lat 1205.114 ms stddev 4898.659 2024-09-16T01:24:57.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:56 smithi078 ceph-mon[30937]: pgmap v444: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 282 KiB/s wr, 43 op/s 2024-09-16T01:24:57.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:56 smithi193 ceph-mon[35701]: pgmap v444: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 282 KiB/s wr, 43 op/s 2024-09-16T01:24:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:56 smithi148 ceph-mon[35030]: pgmap v444: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 282 KiB/s wr, 43 op/s 2024-09-16T01:24:59.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:24:58 smithi078 ceph-mon[30937]: pgmap v445: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 671 KiB/s wr, 90 op/s 2024-09-16T01:24:59.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:24:58 smithi193 ceph-mon[35701]: pgmap v445: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 671 KiB/s wr, 90 op/s 2024-09-16T01:24:59.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:24:58 smithi148 ceph-mon[35030]: pgmap v445: 129 pgs: 129 active+clean; 2.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 671 KiB/s wr, 90 op/s 2024-09-16T01:25:00.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 825.0 s, 800.0 tps, lat 121.287 ms stddev 149.271 2024-09-16T01:25:01.187 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:00 smithi148 ceph-mon[35030]: pgmap v446: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-09-16T01:25:01.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:00 smithi078 ceph-mon[30937]: pgmap v446: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-09-16T01:25:01.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:00 smithi193 ceph-mon[35701]: pgmap v446: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.4 MiB/s wr, 210 op/s 2024-09-16T01:25:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.348+0000 7f69ad782640 1 -- 172.21.15.78:0/3933798551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81091f0 msgr2=0x7f69a8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.348+0000 7f69ad782640 1 --2- 172.21.15.78:0/3933798551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81091f0 0x7f69a8110250 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f69a001acb0 tx=0x7f69a003b920 comp rx=0 tx=0).stop 2024-09-16T01:25:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.350+0000 7f69ad782640 1 -- 172.21.15.78:0/3933798551 shutdown_connections 2024-09-16T01:25:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.350+0000 7f69ad782640 1 --2- 172.21.15.78:0/3933798551 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a811b140 0x7f69a811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.350+0000 7f69ad782640 1 --2- 172.21.15.78:0/3933798551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69a81095d0 0x7f69a8109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.350+0000 7f69ad782640 1 --2- 172.21.15.78:0/3933798551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81091f0 0x7f69a8110250 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.350+0000 7f69ad782640 1 -- 172.21.15.78:0/3933798551 >> 172.21.15.78:0/3933798551 conn(0x7f69a806cbf0 msgr2=0x7f69a806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.357+0000 7f69ad782640 1 -- 172.21.15.78:0/3933798551 shutdown_connections 2024-09-16T01:25:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.361+0000 7f69ad782640 1 -- 172.21.15.78:0/3933798551 wait complete. 2024-09-16T01:25:02.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.361+0000 7f69ad782640 1 Processor -- start 2024-09-16T01:25:02.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 -- start start 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a81091f0 0x7f69a811a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69a811b140 0x7f69a8116f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f69a8113360 con 0x7f69a81095d0 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69a81131e0 con 0x7f69a81091f0 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.365+0000 7f69ad782640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69a81134e0 con 0x7f69a811b140 2024-09-16T01:25:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.366+0000 7f69a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.366+0000 7f69a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58834/0 (socket says 172.21.15.78:58834) 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.366+0000 7f69a77fe640 1 -- 172.21.15.78:0/2489056868 learned_addr learned my addr 172.21.15.78:0/2489056868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a7fff640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a81091f0 0x7f69a811a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a77fe640 1 -- 172.21.15.78:0/2489056868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69a811b140 msgr2=0x7f69a8116f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69a811b140 0x7f69a8116f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a77fe640 1 -- 172.21.15.78:0/2489056868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a81091f0 msgr2=0x7f69a811a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a81091f0 0x7f69a811a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.367+0000 7f69a77fe640 1 -- 172.21.15.78:0/2489056868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a81177c0 con 0x7f69a81095d0 2024-09-16T01:25:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.368+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f699800e9f0 tx=0x7f699800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.368+0000 7f69a57fa640 1 -- 172.21.15.78:0/2489056868 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699800cd80 con 0x7f69a81095d0 2024-09-16T01:25:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.368+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a81aae90 con 0x7f69a81095d0 2024-09-16T01:25:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.368+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a81ab3a0 con 0x7f69a81095d0 2024-09-16T01:25:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.368+0000 7f69a57fa640 1 -- 172.21.15.78:0/2489056868 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6998010910 con 0x7f69a81095d0 2024-09-16T01:25:02.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.369+0000 7f69a57fa640 1 -- 172.21.15.78:0/2489056868 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998018a10 con 0x7f69a81095d0 2024-09-16T01:25:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.372+0000 7f69b1855740 1 Processor -- start 2024-09-16T01:25:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.372+0000 7f69b1855740 1 -- start start 2024-09-16T01:25:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.373+0000 7f69a57fa640 1 -- 172.21.15.78:0/2489056868 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6998018bb0 con 0x7f69a81095d0 2024-09-16T01:25:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.374+0000 7f69a57fa640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 0x7f69900a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69a57fa640 1 -- 172.21.15.78:0/2489056868 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f69980d03c0 con 0x7f69a81095d0 2024-09-16T01:25:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01ba940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5578d01142f0 con 0x5578d01bae80 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5578d00bc270 con 0x5578d01afcc0 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69b1855740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578d01102b0 con 0x5578d01b05e0 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:39986/0 (socket says 172.21.15.78:39986) 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.375+0000 7f69a7fff640 1 -- 172.21.15.78:0/3732551447 learned_addr learned my addr 172.21.15.78:0/3732551447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.377+0000 7f69a77fe640 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01ba940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.378+0000 7f69a77fe640 1 -- 172.21.15.78:0/3732551447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 msgr2=0x5578d01b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.380+0000 7f69acf81640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 0x7f69900a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.381+0000 7f69a77fe640 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.381+0000 7f69a77fe640 1 -- 172.21.15.78:0/3732551447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 msgr2=0x5578d01bd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.381+0000 7f69a77fe640 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.381+0000 7f69a77fe640 1 -- 172.21.15.78:0/3732551447 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d00bb340 con 0x5578d01b05e0 2024-09-16T01:25:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.382+0000 7f69acf81640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 0x7f69900a59c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5578d01ad5b0 tx=0x7f699c006cd0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.383+0000 7f69a77fe640 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01ba940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f69980cd700 tx=0x7f6998018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb56711362dcb93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.389+0000 7f698e7fc640 1 -- 172.21.15.78:0/3732551447 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980a16a0 con 0x5578d01b05e0 2024-09-16T01:25:02.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.389+0000 7f698e7fc640 1 -- 172.21.15.78:0/3732551447 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69980cde50 con 0x5578d01b05e0 2024-09-16T01:25:02.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.390+0000 7f69b1855740 1 -- 172.21.15.78:0/3732551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 msgr2=0x5578d01ba940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.390+0000 7f69b1855740 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01ba940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f69980cd700 tx=0x7f6998018420 comp rx=0 tx=0).stop 2024-09-16T01:25:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.397+0000 7f69b1855740 1 -- 172.21.15.78:0/3732551447 shutdown_connections 2024-09-16T01:25:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.397+0000 7f69b1855740 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.397+0000 7f69b1855740 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01ba940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.398+0000 7f69b1855740 1 --2- 172.21.15.78:0/3732551447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.398+0000 7f69b1855740 1 -- 172.21.15.78:0/3732551447 >> 172.21.15.78:0/3732551447 conn(0x5578d01ac230 msgr2=0x5578d01ac610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.405+0000 7f69b1855740 1 -- 172.21.15.78:0/3732551447 shutdown_connections 2024-09-16T01:25:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.406+0000 7f69b1855740 1 -- 172.21.15.78:0/3732551447 wait complete. 2024-09-16T01:25:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 -- start start 2024-09-16T01:25:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 Processor -- start 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b05e0 0x5578d01b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01bae80 0x5578d01b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5578d01bdaa0 con 0x5578d01b05e0 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5578d009d290 con 0x5578d01afcc0 2024-09-16T01:25:02.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.407+0000 7f69b1855740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578d01bdc20 con 0x5578d01bae80 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69b1855740 10 client.0 _ll_register_callbacks cb 0x5578d00ce140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:40000/0 (socket says 172.21.15.78:40000) 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 -- 172.21.15.78:0/402898216 learned_addr learned my addr 172.21.15.78:0/402898216 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69acf81640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01bae80 0x5578d01b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a77fe640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b05e0 0x5578d01b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 -- 172.21.15.78:0/402898216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01bae80 msgr2=0x5578d01b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01bae80 0x5578d01b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 -- 172.21.15.78:0/402898216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b05e0 msgr2=0x5578d01b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b05e0 0x5578d01b82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.408+0000 7f69a7fff640 1 -- 172.21.15.78:0/402898216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578cfe27210 con 0x5578d01afcc0 2024-09-16T01:25:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.410+0000 7f69a7fff640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69a00408c0 tx=0x7f69a0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a237043b0b98f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.411+0000 7f69877fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:25:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.411+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a005a060 con 0x5578d01afcc0 2024-09-16T01:25:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.411+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69980cde50 con 0x5578d01afcc0 2024-09-16T01:25:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.411+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f69a0040920 con 0x5578d01afcc0 2024-09-16T01:25:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.411+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69a0045050 con 0x5578d01afcc0 2024-09-16T01:25:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.415+0000 7f69b1855740 10 client.26258 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.415+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578cfef3620 con 0x5578d01afcc0 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69980cde50 con 0x5578d01afcc0 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f69a0054070 con 0x5578d01afcc0 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 10 client.26258 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 10 client.26258 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 20 client.26258 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d01cb330 0x5578d01cd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 4 client.26258 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.418+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d0121a20 con 0x5578d01cb330 2024-09-16T01:25:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.421+0000 7f69a77fe640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d01cb330 0x5578d01cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.430+0000 7f69877fe640 10 client.26258 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.430+0000 7f69a77fe640 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d01cb330 0x5578d01cd790 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3676bbba80636e93 server_cookie=4c9cce4162e0789e in_seq=0 out_seq=0 2024-09-16T01:25:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.435+0000 7f69877fe640 1 -- 172.21.15.78:0/402898216 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5578d0121a20 con 0x5578d01cb330 2024-09-16T01:25:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.435+0000 7f69877fe640 10 client.26258 handle_command_reply: tid=0 2024-09-16T01:25:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.435+0000 7f69b1855740 1 client.26258 shutdown 2024-09-16T01:25:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.438+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 msgr2=0x5578d01b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.438+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69a00408c0 tx=0x7f69a0003ba0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 shutdown_connections 2024-09-16T01:25:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d01cb330 0x5578d01cd790 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01bae80 0x5578d01b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b05e0 0x5578d01b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 --2- 172.21.15.78:0/402898216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01afcc0 0x5578d01b7d90 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69a00408c0 tx=0x7f69a0003ba0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 >> 172.21.15.78:0/402898216 conn(0x5578d01ac230 msgr2=0x5578d01b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.439+0000 7f69877fe640 0 client.26258 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.443+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 shutdown_connections 2024-09-16T01:25:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.444+0000 7f69b1855740 1 -- 172.21.15.78:0/402898216 wait complete. 2024-09-16T01:25:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.447+0000 7f69b1855740 20 client.26258 trim_cache size 0 max 16384 2024-09-16T01:25:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69b1855740 1 Processor -- start 2024-09-16T01:25:02.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69b1855740 1 -- start start 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69b1855740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01b8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69b1855740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 0x5578d01ab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69b1855740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 0x5578d01d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01b8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01b8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:48296/0 (socket says 172.21.15.78:48296) 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.476+0000 7f69acf81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 0x5578d01d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69acf81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 0x5578d01d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58850/0 (socket says 172.21.15.78:58850) 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69a7fff640 1 -- 172.21.15.78:0/2003850030 learned_addr learned my addr 172.21.15.78:0/2003850030 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6994012970 con 0x5578d01b6e60 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6994010240 con 0x5578d01bae80 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 0x5578d01ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.477+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f699400e0b0 con 0x5578d01b05e0 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.478+0000 7f69a77fe640 1 -- 172.21.15.78:0/2003850030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 msgr2=0x5578d01b8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.478+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01b8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.478+0000 7f69a77fe640 1 -- 172.21.15.78:0/2003850030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 msgr2=0x5578d01d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.478+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 0x5578d01d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.478+0000 7f69a77fe640 1 -- 172.21.15.78:0/2003850030 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d01ba8e0 con 0x5578d01bae80 2024-09-16T01:25:02.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.479+0000 7f69a77fe640 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 0x5578d01ab150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f699809b5a0 tx=0x7f6998000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cf23930514ac60d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.479+0000 7f698dffb640 1 -- 172.21.15.78:0/2003850030 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699809b5f0 con 0x5578d01bae80 2024-09-16T01:25:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.479+0000 7f698dffb640 1 -- 172.21.15.78:0/2003850030 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69980cde50 con 0x5578d01bae80 2024-09-16T01:25:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.479+0000 7f698dffb640 1 -- 172.21.15.78:0/2003850030 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980a13e0 con 0x5578d01bae80 2024-09-16T01:25:02.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.481+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 msgr2=0x5578d01ab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.481+0000 7f69b1855740 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 0x5578d01ab150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f699809b5a0 tx=0x7f6998000bc0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.484+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 shutdown_connections 2024-09-16T01:25:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.484+0000 7f69b1855740 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01b6e60 0x5578d01d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.484+0000 7f69b1855740 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01bae80 0x5578d01ab150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.484+0000 7f69b1855740 1 --2- 172.21.15.78:0/2003850030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01b8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.484+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 >> 172.21.15.78:0/2003850030 conn(0x5578d01ac230 msgr2=0x5578d01aaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.489+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 shutdown_connections 2024-09-16T01:25:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.493+0000 7f69b1855740 1 -- 172.21.15.78:0/2003850030 wait complete. 2024-09-16T01:25:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.493+0000 7f69b1855740 1 -- start start 2024-09-16T01:25:02.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.493+0000 7f69b1855740 1 Processor -- start 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01c3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 0x5578d01be6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f699400e700 con 0x5578d01bae80 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f699400fb70 con 0x5578d01b6e60 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f699400edd0 con 0x5578d01b05e0 2024-09-16T01:25:02.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.495+0000 7f69b1855740 10 client.0 _ll_register_callbacks cb 0x7f69a001ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.497+0000 7f69a77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 0x5578d01be6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.497+0000 7f69a7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.497+0000 7f69acf81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.497+0000 7f69acf81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58862/0 (socket says 172.21.15.78:58862) 2024-09-16T01:25:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.497+0000 7f69acf81640 1 -- 172.21.15.78:0/4279126990 learned_addr learned my addr 172.21.15.78:0/4279126990 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.498+0000 7f69a77fe640 1 -- 172.21.15.78:0/4279126990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 msgr2=0x5578d01c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.498+0000 7f69a77fe640 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01c3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.498+0000 7f69a77fe640 1 -- 172.21.15.78:0/4279126990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 msgr2=0x5578d01bebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.498+0000 7f69a77fe640 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.498+0000 7f69a77fe640 1 -- 172.21.15.78:0/4279126990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d01bf130 con 0x5578d01b6e60 2024-09-16T01:25:02.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.499+0000 7f69a77fe640 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 0x5578d01be6b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x5578d01b05b0 tx=0x7f6998018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb885cb99ce11b62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.501+0000 7f69877fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:25:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.501+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980b1070 con 0x5578d01b6e60 2024-09-16T01:25:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.501+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69980cde50 con 0x5578d01b6e60 2024-09-16T01:25:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.501+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9525+0+0 (secure 0 0 0) 0x7f699809c8e0 con 0x5578d01b6e60 2024-09-16T01:25:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.504+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69980a16a0 con 0x5578d01b6e60 2024-09-16T01:25:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.504+0000 7f69b1855740 10 client.26264 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:02.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.504+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578d0235d90 con 0x5578d01b6e60 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.507+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69980cde50 con 0x5578d01b6e60 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.507+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6998010430 con 0x5578d01b6e60 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 10 client.26264 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 20 client.26264 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d0235fc0 0x5578d02383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:02.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 4 client.26264 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:25:02.513 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.508+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578d011bd70 con 0x5578d0235fc0 2024-09-16T01:25:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.512+0000 7f69a7fff640 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d0235fc0 0x5578d02383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.517+0000 7f69a7fff640 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d0235fc0 0x5578d02383b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=584a889ca975358c server_cookie=47a43b8bbff2e8e8 in_seq=0 out_seq=0 2024-09-16T01:25:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.517+0000 7f69877fe640 10 client.26264 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69877fe640 1 -- 172.21.15.78:0/4279126990 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578d011bd70 con 0x5578d0235fc0 2024-09-16T01:25:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69877fe640 10 client.26264 handle_command_reply: tid=0 2024-09-16T01:25:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 client.26264 shutdown 2024-09-16T01:25:02.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 msgr2=0x5578d01be6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 0x5578d01be6b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x5578d01b05b0 tx=0x7f6998018420 comp rx=0 tx=0).stop 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 shutdown_connections 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5578d0235fc0 0x5578d02383b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578d01bae80 0x5578d01bebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5578d01b6e60 0x5578d01be6b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 --2- 172.21.15.78:0/4279126990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578d01b05e0 0x5578d01c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 >> 172.21.15.78:0/4279126990 conn(0x5578d01ac230 msgr2=0x5578d01ab260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69877fe640 0 client.26264 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.635+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 shutdown_connections 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.636+0000 7f69b1855740 1 -- 172.21.15.78:0/4279126990 wait complete. 2024-09-16T01:25:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.636+0000 7f69b1855740 20 client.26264 trim_cache size 0 max 16384 2024-09-16T01:25:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:25:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:25:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:25:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 msgr2=0x7f69900a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 0x7f69900a59c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5578d01ad5b0 tx=0x7f699c006cd0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 msgr2=0x7f69a8116a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f699800e9f0 tx=0x7f699800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 shutdown_connections 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f69900a3500 0x7f69900a59c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69a811b140 0x7f69a8116f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a81095d0 0x7f69a8116a50 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 --2- 172.21.15.78:0/2489056868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69a81091f0 0x7f69a811a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 >> 172.21.15.78:0/2489056868 conn(0x7f69a806cbf0 msgr2=0x7f69a8070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 shutdown_connections 2024-09-16T01:25:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:02.639+0000 7f69ad782640 1 -- 172.21.15.78:0/2489056868 wait complete. 2024-09-16T01:25:03.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:02 smithi078 ceph-mon[30937]: pgmap v447: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 MiB/s wr, 354 op/s 2024-09-16T01:25:03.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:02 smithi193 ceph-mon[35701]: pgmap v447: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 MiB/s wr, 354 op/s 2024-09-16T01:25:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:02 smithi148 ceph-mon[35030]: pgmap v447: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 MiB/s wr, 354 op/s 2024-09-16T01:25:04.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:03 smithi078 ceph-mon[30937]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T01:25:04.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:03 smithi193 ceph-mon[35701]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T01:25:04.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:03 smithi148 ceph-mon[35030]: osdmap e112: 12 total, 12 up, 12 in 2024-09-16T01:25:05.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:04 smithi078 ceph-mon[30937]: pgmap v449: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 499 op/s 2024-09-16T01:25:05.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:04 smithi193 ceph-mon[35701]: pgmap v449: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 499 op/s 2024-09-16T01:25:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:04 smithi148 ceph-mon[35030]: pgmap v449: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 4.7 MiB/s wr, 499 op/s 2024-09-16T01:25:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 830.0 s, 507.4 tps, lat 203.339 ms stddev 262.919 2024-09-16T01:25:07.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:06 smithi078 ceph-mon[30937]: pgmap v450: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 650 op/s 2024-09-16T01:25:07.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:06 smithi193 ceph-mon[35701]: pgmap v450: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 650 op/s 2024-09-16T01:25:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:06 smithi148 ceph-mon[35030]: pgmap v450: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 650 op/s 2024-09-16T01:25:09.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:08 smithi078 ceph-mon[30937]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 664 op/s 2024-09-16T01:25:09.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:08 smithi193 ceph-mon[35701]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 664 op/s 2024-09-16T01:25:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:08 smithi148 ceph-mon[35030]: pgmap v451: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 2.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 664 op/s 2024-09-16T01:25:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 835.0 s, 536.8 tps, lat 185.959 ms stddev 218.396 2024-09-16T01:25:11.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:10 smithi078 ceph-mon[30937]: pgmap v452: 129 pgs: 129 active+clean; 2.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.7 MiB/s wr, 635 op/s 2024-09-16T01:25:11.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:10 smithi078 ceph-mon[30937]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T01:25:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:10 smithi193 ceph-mon[35701]: pgmap v452: 129 pgs: 129 active+clean; 2.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.7 MiB/s wr, 635 op/s 2024-09-16T01:25:11.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:10 smithi193 ceph-mon[35701]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T01:25:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:10 smithi148 ceph-mon[35030]: pgmap v452: 129 pgs: 129 active+clean; 2.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 5.7 MiB/s wr, 635 op/s 2024-09-16T01:25:11.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:10 smithi148 ceph-mon[35030]: osdmap e113: 12 total, 12 up, 12 in 2024-09-16T01:25:13.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:12 smithi078 ceph-mon[30937]: pgmap v454: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:25:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:25:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:25:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:25:13.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:12 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:25:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:12 smithi193 ceph-mon[35701]: pgmap v454: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:25:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:25:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:25:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:25:13.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:12 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:25:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:12 smithi148 ceph-mon[35030]: pgmap v454: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 5.8 MiB/s wr, 640 op/s 2024-09-16T01:25:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:25:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:25:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:25:13.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:12 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:25:15.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:14 smithi078 ceph-mon[30937]: pgmap v455: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 5.1 MiB/s wr, 543 op/s 2024-09-16T01:25:15.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:14 smithi193 ceph-mon[35701]: pgmap v455: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 5.1 MiB/s wr, 543 op/s 2024-09-16T01:25:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:14 smithi148 ceph-mon[35030]: pgmap v455: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 5.1 MiB/s wr, 543 op/s 2024-09-16T01:25:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.801+0000 7f6dde8af640 1 -- 172.21.15.78:0/1556376484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd8112a50 msgr2=0x7f6dd8114e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.801+0000 7f6dde8af640 1 --2- 172.21.15.78:0/1556376484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd8112a50 0x7f6dd8114e40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009be0 tx=0x7f6dd002a790 comp rx=0 tx=0).stop 2024-09-16T01:25:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.802+0000 7f6dde8af640 1 -- 172.21.15.78:0/1556376484 shutdown_connections 2024-09-16T01:25:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.802+0000 7f6dde8af640 1 --2- 172.21.15.78:0/1556376484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd8112a50 0x7f6dd8114e40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.802+0000 7f6dde8af640 1 --2- 172.21.15.78:0/1556376484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8105fc0 0x7f6dd8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.802+0000 7f6dde8af640 1 --2- 172.21.15.78:0/1556376484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd81056a0 0x7f6dd8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.802+0000 7f6dde8af640 1 -- 172.21.15.78:0/1556376484 >> 172.21.15.78:0/1556376484 conn(0x7f6dd8075ec0 msgr2=0x7f6dd80762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.804+0000 7f6dde8af640 1 -- 172.21.15.78:0/1556376484 shutdown_connections 2024-09-16T01:25:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.805+0000 7f6dde8af640 1 -- 172.21.15.78:0/1556376484 wait complete. 2024-09-16T01:25:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.806+0000 7f6dde8af640 1 Processor -- start 2024-09-16T01:25:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.807+0000 7f6dde8af640 1 -- start start 2024-09-16T01:25:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.807+0000 7f6dde8af640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.807+0000 7f6dde8af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8105fc0 0x7f6dd80fee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59398/0 (socket says 172.21.15.78:59398) 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dd7fff640 1 -- 172.21.15.78:0/2873192317 learned_addr learned my addr 172.21.15.78:0/2873192317 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 0x7f6dd80ff3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8105fc0 0x7f6dd80fee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd806c660 con 0x7f6dd8105fc0 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 0x7f6dd80ff3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd806c4e0 con 0x7f6dd81056a0 2024-09-16T01:25:15.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.808+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd806c7e0 con 0x7f6dd8112a50 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6dd7fff640 1 -- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 msgr2=0x7f6dd80ff3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 0x7f6dd80ff3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6dd7fff640 1 -- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8105fc0 msgr2=0x7f6dd80fee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8105fc0 0x7f6dd80fee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6dd7fff640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd80ffb50 con 0x7f6dd81056a0 2024-09-16T01:25:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.812+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 0x7f6dd80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:25:15.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.814+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00e9c0 tx=0x7f6dcc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.816+0000 7f6dd57fa640 1 -- 172.21.15.78:0/2873192317 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00cc40 con 0x7f6dd81056a0 2024-09-16T01:25:15.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.816+0000 7f6dd57fa640 1 -- 172.21.15.78:0/2873192317 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dcc00cde0 con 0x7f6dd81056a0 2024-09-16T01:25:15.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.816+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81b37e0 con 0x7f6dd81056a0 2024-09-16T01:25:15.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.816+0000 7f6dd57fa640 1 -- 172.21.15.78:0/2873192317 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc010430 con 0x7f6dd81056a0 2024-09-16T01:25:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.817+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd81b3cd0 con 0x7f6dd81056a0 2024-09-16T01:25:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.823+0000 7f6de1642740 1 Processor -- start 2024-09-16T01:25:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.823+0000 7f6de1642740 1 -- start start 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.824+0000 7f6de1642740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a17418ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.824+0000 7f6de1642740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a17418d410 0x55a174197770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.824+0000 7f6de1642740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.825+0000 7f6de1642740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a1740f12f0 con 0x55a17418caf0 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.825+0000 7f6de1642740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a174099270 con 0x55a17418d410 2024-09-16T01:25:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.825+0000 7f6de1642740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a1740ed2b0 con 0x55a174197cb0 2024-09-16T01:25:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42596/0 (socket says 172.21.15.78:42596) 2024-09-16T01:25:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 -- 172.21.15.78:0/1979450104 learned_addr learned my addr 172.21.15.78:0/1979450104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 -- 172.21.15.78:0/1979450104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a17418d410 msgr2=0x55a174197770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a17418d410 0x55a174197770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 -- 172.21.15.78:0/1979450104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 msgr2=0x55a17418ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a17418ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.826+0000 7f6dd77fe640 1 -- 172.21.15.78:0/1979450104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a174098340 con 0x55a174197cb0 2024-09-16T01:25:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.827+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800b9c0 tx=0x7f6dc800be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc30e7afecb4d71a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.828+0000 7f6dbe7fc640 1 -- 172.21.15.78:0/1979450104 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800de50 con 0x55a174197cb0 2024-09-16T01:25:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.828+0000 7f6dbe7fc640 1 -- 172.21.15.78:0/1979450104 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dc8017900 con 0x55a174197cb0 2024-09-16T01:25:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.828+0000 7f6dbe7fc640 1 -- 172.21.15.78:0/1979450104 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800fac0 con 0x55a174197cb0 2024-09-16T01:25:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.829+0000 7f6de1642740 1 -- 172.21.15.78:0/1979450104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 msgr2=0x55a17419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.829+0000 7f6de1642740 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800b9c0 tx=0x7f6dc800be80 comp rx=0 tx=0).stop 2024-09-16T01:25:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.828+0000 7f6dd57fa640 1 -- 172.21.15.78:0/2873192317 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6dcc005000 con 0x7f6dd81056a0 2024-09-16T01:25:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.830+0000 7f6dd57fa640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 0x7f6dc00a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6dd57fa640 1 -- 172.21.15.78:0/2873192317 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f6dcc0cfdf0 con 0x7f6dd81056a0 2024-09-16T01:25:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6de1642740 1 -- 172.21.15.78:0/1979450104 shutdown_connections 2024-09-16T01:25:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6de1642740 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174197cb0 0x55a17419a0a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6de1642740 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a17418d410 0x55a174197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6de1642740 1 --2- 172.21.15.78:0/1979450104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a17418ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6de1642740 1 -- 172.21.15.78:0/1979450104 >> 172.21.15.78:0/1979450104 conn(0x55a174189060 msgr2=0x55a174189440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.831+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 0x7f6dc00a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.832+0000 7f6de1642740 1 -- 172.21.15.78:0/1979450104 shutdown_connections 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 -- 172.21.15.78:0/1979450104 wait complete. 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 -- start start 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 Processor -- start 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a174194d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174195240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.833+0000 7f6de1642740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.834+0000 7f6de1642740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a17419a880 con 0x55a17418caf0 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.834+0000 7f6de1642740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a17407a290 con 0x55a174197cb0 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.834+0000 7f6de1642740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a17419aa00 con 0x55a174195780 2024-09-16T01:25:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.834+0000 7f6de1642740 10 client.0 _ll_register_callbacks cb 0x55a1740ab140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.835+0000 7f6ddce4c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.835+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 0x7f6dc00a5b50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd0009840 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.835+0000 7f6ddce4c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:42606/0 (socket says 172.21.15.78:42606) 2024-09-16T01:25:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.835+0000 7f6ddce4c640 1 -- 172.21.15.78:0/2245964697 learned_addr learned my addr 172.21.15.78:0/2245964697 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a174194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6ddce4c640 1 -- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 msgr2=0x55a174195240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:25:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174195240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6ddce4c640 1 -- 172.21.15.78:0/2245964697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 msgr2=0x55a174194d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a174194d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.836+0000 7f6ddce4c640 1 -- 172.21.15.78:0/2245964697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a173e04210 con 0x55a174195780 2024-09-16T01:25:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.839+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0033000 tx=0x7f6dd002a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da38282b50cd4f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.840+0000 7f6dbbff7640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:25:15.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.840+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd004e020 con 0x55a174195780 2024-09-16T01:25:15.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.840+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc8017900 con 0x55a174195780 2024-09-16T01:25:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.840+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f6dd00045c0 con 0x55a174195780 2024-09-16T01:25:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.841+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6dd0036040 con 0x55a174195780 2024-09-16T01:25:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.841+0000 7f6de1642740 10 client.26305 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.841+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a173ed0620 con 0x55a174195780 2024-09-16T01:25:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dc8017900 con 0x55a174195780 2024-09-16T01:25:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6dd005b020 con 0x55a174195780 2024-09-16T01:25:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 10 client.26305 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:15.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 20 client.26305 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1741b9640 0x55a1741bbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 4 client.26305 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.842+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1740fea20 con 0x55a1741b9640 2024-09-16T01:25:15.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.843+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1741b9640 0x55a1741bbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.844+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1741b9640 0x55a1741bbac0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b2be6eb321a639 server_cookie=6a820724533f31e6 in_seq=0 out_seq=0 2024-09-16T01:25:15.845 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.844+0000 7f6dbbff7640 10 client.26305 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.845+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2245964697 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a1740fea20 con 0x55a1741b9640 2024-09-16T01:25:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.845+0000 7f6dbbff7640 10 client.26305 handle_command_reply: tid=0 2024-09-16T01:25:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.845+0000 7f6de1642740 1 client.26305 shutdown 2024-09-16T01:25:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.845+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 msgr2=0x55a1741b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.846 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.845+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0033000 tx=0x7f6dd002a790 comp rx=0 tx=0).stop 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 shutdown_connections 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1741b9640 0x55a1741bbac0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a174195780 0x55a1741b4d70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 --2- 172.21.15.78:0/2245964697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a17418caf0 0x55a174194d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.848 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 >> 172.21.15.78:0/2245964697 conn(0x55a174189060 msgr2=0x55a17418edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6dbbff7640 0 client.26305 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 shutdown_connections 2024-09-16T01:25:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 1 -- 172.21.15.78:0/2245964697 wait complete. 2024-09-16T01:25:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.847+0000 7f6de1642740 20 client.26305 trim_cache size 0 max 16384 2024-09-16T01:25:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.872+0000 7f6de1642740 1 Processor -- start 2024-09-16T01:25:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.872+0000 7f6de1642740 1 -- start start 2024-09-16T01:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174193b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81b80c0 con 0x55a174195780 2024-09-16T01:25:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81b5990 con 0x55a174197cb0 2024-09-16T01:25:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6de1642740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd8105a80 con 0x55a17418caf0 2024-09-16T01:25:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.873+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45602/0 (socket says 172.21.15.78:45602) 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 -- 172.21.15.78:0/3475224125 learned_addr learned my addr 172.21.15.78:0/3475224125 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 -- 172.21.15.78:0/3475224125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 msgr2=0x55a1741b4be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b4be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 -- 172.21.15.78:0/3475224125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 msgr2=0x55a174193b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174193b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 -- 172.21.15.78:0/3475224125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a174197850 con 0x55a174195780 2024-09-16T01:25:15.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.874+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8019040 tx=0x7f6dc800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f11340968b504a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.882+0000 7f6dbdffb640 1 -- 172.21.15.78:0/3475224125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8025020 con 0x55a174195780 2024-09-16T01:25:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.882+0000 7f6dbdffb640 1 -- 172.21.15.78:0/3475224125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dc800de90 con 0x55a174195780 2024-09-16T01:25:15.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.882+0000 7f6dbdffb640 1 -- 172.21.15.78:0/3475224125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc80206e0 con 0x55a174195780 2024-09-16T01:25:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.884+0000 7f6de1642740 1 -- 172.21.15.78:0/3475224125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 msgr2=0x55a174193610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.884+0000 7f6de1642740 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8019040 tx=0x7f6dc800dd20 comp rx=0 tx=0).stop 2024-09-16T01:25:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.888+0000 7f6de1642740 1 -- 172.21.15.78:0/3475224125 shutdown_connections 2024-09-16T01:25:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.888+0000 7f6de1642740 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174197cb0 0x55a174193b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.888+0000 7f6de1642740 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174195780 0x55a174193610 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.888+0000 7f6de1642740 1 --2- 172.21.15.78:0/3475224125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.888+0000 7f6de1642740 1 -- 172.21.15.78:0/3475224125 >> 172.21.15.78:0/3475224125 conn(0x55a174189060 msgr2=0x55a174194cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.894+0000 7f6de1642740 1 -- 172.21.15.78:0/3475224125 shutdown_connections 2024-09-16T01:25:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.900+0000 7f6de1642740 1 -- 172.21.15.78:0/3475224125 wait complete. 2024-09-16T01:25:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.900+0000 7f6de1642740 1 -- start start 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.900+0000 7f6de1642740 1 Processor -- start 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 0x55a1741ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81b3f60 con 0x55a174197cb0 2024-09-16T01:25:15.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81b52c0 con 0x55a174195780 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81b4520 con 0x55a17418caf0 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6de1642740 10 client.0 _ll_register_callbacks cb 0x7f6dcc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.901+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.902+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.902+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:59432/0 (socket says 172.21.15.78:59432) 2024-09-16T01:25:15.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.902+0000 7f6dd77fe640 1 -- 172.21.15.78:0/2870106740 learned_addr learned my addr 172.21.15.78:0/2870106740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 0x55a1741ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6dd77fe640 1 -- 172.21.15.78:0/2870106740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 msgr2=0x55a1741b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6dd77fe640 1 -- 172.21.15.78:0/2870106740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 msgr2=0x55a1741ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 0x55a1741ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6dd77fe640 1 -- 172.21.15.78:0/2870106740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1741acb20 con 0x55a174195780 2024-09-16T01:25:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.904+0000 7f6ddce4c640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 0x55a1741ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:25:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.909+0000 7f6dd77fe640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800bfd0 tx=0x7f6dc8005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77afe4a79b17106 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.910+0000 7f6dbbff7640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:25:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.914+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc802e070 con 0x55a174195780 2024-09-16T01:25:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.914+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f6dc8005520 con 0x55a174195780 2024-09-16T01:25:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.915+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc800de90 con 0x55a174195780 2024-09-16T01:25:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.918+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6dc8038050 con 0x55a174195780 2024-09-16T01:25:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.918+0000 7f6de1642740 10 client.26273 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.918+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a174223ee0 con 0x55a174195780 2024-09-16T01:25:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dc800de90 con 0x55a174195780 2024-09-16T01:25:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6dc8033520 con 0x55a174195780 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6de1642740 10 client.26273 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6de1642740 10 client.26273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6de1642740 20 client.26273 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.920+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1742242b0 0x55a1742266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.921+0000 7f6de1642740 4 client.26273 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.921+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a1740f8d70 con 0x55a1742242b0 2024-09-16T01:25:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.921+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1742242b0 0x55a1742266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.932+0000 7f6dd7fff640 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1742242b0 0x55a1742266a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95f82ebf59d678e server_cookie=c205c8b30b62f440 in_seq=0 out_seq=0 2024-09-16T01:25:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.933+0000 7f6dbbff7640 10 client.26273 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:15.938 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 840.0 s, 510.1 tps, lat 196.181 ms stddev 231.215 2024-09-16T01:25:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.963+0000 7f6dbbff7640 1 -- 172.21.15.78:0/2870106740 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a1740f8d70 con 0x55a1742242b0 2024-09-16T01:25:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.963+0000 7f6dbbff7640 10 client.26273 handle_command_reply: tid=0 2024-09-16T01:25:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.963+0000 7f6de1642740 1 client.26273 shutdown 2024-09-16T01:25:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.964+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 msgr2=0x55a1741b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.964+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800bfd0 tx=0x7f6dc8005020 comp rx=0 tx=0).stop 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 shutdown_connections 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a1742242b0 0x55a1742266a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a174197cb0 0x55a1741ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a174195780 0x55a1741b1480 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 --2- 172.21.15.78:0/2870106740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a17418caf0 0x55a1741b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 >> 172.21.15.78:0/2870106740 conn(0x55a174189060 msgr2=0x55a1741935d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6dbbff7640 0 client.26273 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.967+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 shutdown_connections 2024-09-16T01:25:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.970+0000 7f6de1642740 1 -- 172.21.15.78:0/2870106740 wait complete. 2024-09-16T01:25:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.970+0000 7f6de1642740 20 client.26273 trim_cache size 0 max 16384 2024-09-16T01:25:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:25:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:25:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:25:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:25:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.971+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 msgr2=0x7f6dc00a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.971+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 0x7f6dc00a5b50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd0009840 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.971+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 msgr2=0x7f6dd8104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.971+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00e9c0 tx=0x7f6dcc00ee80 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 shutdown_connections 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6dc00a3690 0x7f6dc00a5b50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dd8112a50 0x7f6dd80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8105fc0 0x7f6dd80fee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 --2- 172.21.15.78:0/2873192317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6dd81056a0 0x7f6dd8104e20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.972+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 >> 172.21.15.78:0/2873192317 conn(0x7f6dd8075ec0 msgr2=0x7f6dd8111450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.973+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 shutdown_connections 2024-09-16T01:25:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:15.973+0000 7f6dde8af640 1 -- 172.21.15.78:0/2873192317 wait complete. 2024-09-16T01:25:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:16 smithi078 ceph-mon[30937]: pgmap v456: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 496 op/s 2024-09-16T01:25:17.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:16 smithi193 ceph-mon[35701]: pgmap v456: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 496 op/s 2024-09-16T01:25:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:16 smithi148 ceph-mon[35030]: pgmap v456: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 496 op/s 2024-09-16T01:25:19.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:18 smithi078 ceph-mon[30937]: pgmap v457: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.9 MiB/s wr, 498 op/s 2024-09-16T01:25:19.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:18 smithi193 ceph-mon[35701]: pgmap v457: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.9 MiB/s wr, 498 op/s 2024-09-16T01:25:19.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:18 smithi148 ceph-mon[35030]: pgmap v457: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 4.9 MiB/s wr, 498 op/s 2024-09-16T01:25:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 845.0 s, 448.5 tps, lat 219.280 ms stddev 315.617 2024-09-16T01:25:21.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:20 smithi078 ceph-mon[30937]: pgmap v458: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-09-16T01:25:21.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:20 smithi193 ceph-mon[35701]: pgmap v458: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-09-16T01:25:21.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:20 smithi148 ceph-mon[35030]: pgmap v458: 129 pgs: 129 active+clean; 2.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-09-16T01:25:23.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:22 smithi078 ceph-mon[30937]: pgmap v459: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:25:23.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:22 smithi193 ceph-mon[35701]: pgmap v459: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:25:23.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:22 smithi148 ceph-mon[35030]: pgmap v459: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 15 MiB/s wr, 1.22k op/s 2024-09-16T01:25:25.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:24 smithi078 ceph-mon[30937]: pgmap v460: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-09-16T01:25:25.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:24 smithi193 ceph-mon[35701]: pgmap v460: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-09-16T01:25:25.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:24 smithi148 ceph-mon[35030]: pgmap v460: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-09-16T01:25:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 850.0 s, 574.0 tps, lat 173.429 ms stddev 211.074 2024-09-16T01:25:27.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:26 smithi078 ceph-mon[30937]: pgmap v461: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-09-16T01:25:27.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:26 smithi193 ceph-mon[35701]: pgmap v461: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-09-16T01:25:27.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:26 smithi148 ceph-mon[35030]: pgmap v461: 129 pgs: 129 active+clean; 3.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-09-16T01:25:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 -- 172.21.15.78:0/133731659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c0fff60 msgr2=0x7f829c1003a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 --2- 172.21.15.78:0/133731659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c0fff60 0x7f829c1003a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f828c009900 tx=0x7f828c02a410 comp rx=0 tx=0).stop 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 -- 172.21.15.78:0/133731659 shutdown_connections 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 --2- 172.21.15.78:0/133731659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c1008e0 0x7f829c10ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 --2- 172.21.15.78:0/133731659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c0fff60 0x7f829c1003a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 --2- 172.21.15.78:0/133731659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c103aa0 0x7f829c103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 -- 172.21.15.78:0/133731659 >> 172.21.15.78:0/133731659 conn(0x7f829c0770f0 msgr2=0x7f829c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 -- 172.21.15.78:0/133731659 shutdown_connections 2024-09-16T01:25:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.102+0000 7f82a3e77640 1 -- 172.21.15.78:0/133731659 wait complete. 2024-09-16T01:25:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 Processor -- start 2024-09-16T01:25:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 -- start start 2024-09-16T01:25:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c1008e0 0x7f829c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1120a0 con 0x7f829c103aa0 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f829c111f20 con 0x7f829c0fff60 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.103+0000 7f82a3e77640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f829c112220 con 0x7f829c1008e0 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a2414640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a2414640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34646/0 (socket says 172.21.15.78:34646) 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41790/0 (socket says 172.21.15.78:41790) 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a2414640 1 -- 172.21.15.78:0/1568346164 learned_addr learned my addr 172.21.15.78:0/1568346164 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1412640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c1008e0 0x7f829c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 -- 172.21.15.78:0/1568346164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c1008e0 msgr2=0x7f829c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c1008e0 0x7f829c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 -- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 msgr2=0x7f829c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 -- 172.21.15.78:0/1568346164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c19d4e0 con 0x7f829c0fff60 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a2414640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a1c13640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f829800e9f0 tx=0x7f829800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f8292ffd640 1 -- 172.21.15.78:0/1568346164 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829800ccc0 con 0x7f829c0fff60 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c19d7d0 con 0x7f829c0fff60 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f8292ffd640 1 -- 172.21.15.78:0/1568346164 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8298010910 con 0x7f829c0fff60 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f8292ffd640 1 -- 172.21.15.78:0/1568346164 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298018960 con 0x7f829c0fff60 2024-09-16T01:25:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.104+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c101070 con 0x7f829c0fff60 2024-09-16T01:25:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.108+0000 7f82a6c0f740 1 Processor -- start 2024-09-16T01:25:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 -- start start 2024-09-16T01:25:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56364232a2f0 con 0x5636423d94c0 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5636422d2270 con 0x5636423c6330 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5636423262b0 con 0x5636423c5a10 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.109+0000 7f82a1412640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a1412640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423db8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34652/0 (socket says 172.21.15.78:34652) 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a1412640 1 -- 172.21.15.78:0/4127768267 learned_addr learned my addr 172.21.15.78:0/4127768267 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 -- 172.21.15.78:0/4127768267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 msgr2=0x5636423d8f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d8f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 -- 172.21.15.78:0/4127768267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 msgr2=0x5636423db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423db8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 -- 172.21.15.78:0/4127768267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636422d1340 con 0x5636423c5a10 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f82a2414640 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423c5df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f82940049d0 tx=0x7f8294011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b9d2c574920aa4e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f8292ffd640 1 -- 172.21.15.78:0/1568346164 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f8298010ac0 con 0x7f829c0fff60 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f8287fff640 1 -- 172.21.15.78:0/4127768267 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294011310 con 0x5636423c5a10 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f8287fff640 1 -- 172.21.15.78:0/4127768267 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829400f9a0 con 0x5636423c5a10 2024-09-16T01:25:29.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.110+0000 7f8287fff640 1 -- 172.21.15.78:0/4127768267 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294005790 con 0x5636423c5a10 2024-09-16T01:25:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.111+0000 7f82a6c0f740 1 -- 172.21.15.78:0/4127768267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 msgr2=0x5636423c5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.111+0000 7f8292ffd640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 0x7f82880a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.111+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423c5df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f82940049d0 tx=0x7f8294011c90 comp rx=0 tx=0).stop 2024-09-16T01:25:29.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.111+0000 7f82a1c13640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 0x7f82880a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.111+0000 7f8292ffd640 1 -- 172.21.15.78:0/1568346164 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f82980d0430 con 0x7f829c0fff60 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 -- 172.21.15.78:0/4127768267 shutdown_connections 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/4127768267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423c5df0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 -- 172.21.15.78:0/4127768267 >> 172.21.15.78:0/4127768267 conn(0x5636423c2060 msgr2=0x5636423c2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 -- 172.21.15.78:0/4127768267 shutdown_connections 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 -- 172.21.15.78:0/4127768267 wait complete. 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a1c13640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 0x7f82880a5920 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f82980cd7f0 tx=0x7f8298025360 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 -- start start 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.112+0000 7f82a6c0f740 1 Processor -- start 2024-09-16T01:25:29.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423d6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a2414640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a2414640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:50712/0 (socket says 172.21.15.78:50712) 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a1412640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a1412640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41792/0 (socket says 172.21.15.78:41792) 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a2414640 1 -- 172.21.15.78:0/45763736 learned_addr learned my addr 172.21.15.78:0/45763736 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5636423dc090 con 0x5636423d94c0 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5636422b3290 con 0x5636423c6330 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5636423dc210 con 0x5636423c5a10 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.113+0000 7f82a2414640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423d6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 -- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 msgr2=0x5636423d6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 -- 172.21.15.78:0/45763736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 msgr2=0x5636423d6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423d6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56364203d210 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a6c0f740 10 client.0 _ll_register_callbacks cb 0x5636422e4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a2414640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8294005270 tx=0x7f8294005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d4cb0f8d6bb44be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82857fa640 10 client.26320 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f829400f9a0 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82857fa640 1 -- 172.21.15.78:0/45763736 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829402f060 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82857fa640 1 -- 172.21.15.78:0/45763736 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f8294004a40 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82857fa640 1 -- 172.21.15.78:0/45763736 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f829401e040 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a6c0f740 10 client.26320 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.114+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x563642109620 con 0x5636423c5a10 2024-09-16T01:25:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82857fa640 1 -- 172.21.15.78:0/45763736 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8294024070 con 0x5636423c5a10 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 10 client.26320 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 10 client.26320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 20 client.26320 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423e90e0 0x5636423eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 4 client.26320 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.115+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563642337a20 con 0x5636423e90e0 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.116+0000 7f82a1412640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423e90e0 0x5636423eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.116+0000 7f82a1412640 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423e90e0 0x5636423eb4d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ea072b0b515d33 server_cookie=a7e584c37b4b246c in_seq=0 out_seq=0 2024-09-16T01:25:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.116+0000 7f82857fa640 10 client.26320 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82857fa640 1 -- 172.21.15.78:0/45763736 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563642337a20 con 0x5636423e90e0 2024-09-16T01:25:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82857fa640 10 client.26320 handle_command_reply: tid=0 2024-09-16T01:25:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 client.26320 shutdown 2024-09-16T01:25:29.122 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 msgr2=0x5636423d6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8294005270 tx=0x7f8294005370 comp rx=0 tx=0).stop 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 shutdown_connections 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423e90e0 0x5636423eb4d0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423d94c0 0x5636423d6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423d6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/45763736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423c5a10 0x5636423d6320 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 >> 172.21.15.78:0/45763736 conn(0x5636423c2060 msgr2=0x5636423c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82857fa640 0 client.26320 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 shutdown_connections 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.117+0000 7f82a6c0f740 1 -- 172.21.15.78:0/45763736 wait complete. 2024-09-16T01:25:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.118+0000 7f82a6c0f740 20 client.26320 trim_cache size 0 max 16384 2024-09-16T01:25:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.148+0000 7f82a6c0f740 1 Processor -- start 2024-09-16T01:25:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 -- start start 2024-09-16T01:25:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 0x5636423f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423d3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1af820 con 0x5636423c5a10 2024-09-16T01:25:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1ad0f0 con 0x5636423c6330 2024-09-16T01:25:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.149+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1aaf60 con 0x5636423d94c0 2024-09-16T01:25:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.155+0000 7f82a1412640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.160+0000 7f82a1412640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41804/0 (socket says 172.21.15.78:41804) 2024-09-16T01:25:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.160+0000 7f82a1412640 1 -- 172.21.15.78:0/3617823920 learned_addr learned my addr 172.21.15.78:0/3617823920 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.164+0000 7f82a2414640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423d3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.164+0000 7f82a2414640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 0x5636423f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a2414640 1 -- 172.21.15.78:0/3617823920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 msgr2=0x5636423d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a2414640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423d3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a2414640 1 -- 172.21.15.78:0/3617823920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 msgr2=0x5636423f11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a2414640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a2414640 1 -- 172.21.15.78:0/3617823920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636423c6160 con 0x5636423c5a10 2024-09-16T01:25:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.169+0000 7f82a1412640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:25:29.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.174+0000 7f82a2414640 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 0x5636423f0c60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f82940052a0 tx=0x7f8294022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f78df5c533e3e5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.176+0000 7f82877fe640 1 -- 172.21.15.78:0/3617823920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294005340 con 0x5636423c5a10 2024-09-16T01:25:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.176+0000 7f82877fe640 1 -- 172.21.15.78:0/3617823920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829400f9a0 con 0x5636423c5a10 2024-09-16T01:25:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.177+0000 7f82a6c0f740 1 -- 172.21.15.78:0/3617823920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 msgr2=0x5636423f0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.177+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 0x5636423f0c60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f82940052a0 tx=0x7f8294022000 comp rx=0 tx=0).stop 2024-09-16T01:25:29.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.178+0000 7f82a6c0f740 1 -- 172.21.15.78:0/3617823920 shutdown_connections 2024-09-16T01:25:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.178+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423d3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.178+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c6330 0x5636423f11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.178+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/3617823920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c5a10 0x5636423f0c60 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.178+0000 7f82a6c0f740 1 -- 172.21.15.78:0/3617823920 >> 172.21.15.78:0/3617823920 conn(0x5636423c2060 msgr2=0x5636423c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.183+0000 7f82a6c0f740 1 -- 172.21.15.78:0/3617823920 shutdown_connections 2024-09-16T01:25:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.187+0000 7f82a6c0f740 1 -- 172.21.15.78:0/3617823920 wait complete. 2024-09-16T01:25:29.191 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.187+0000 7f82a6c0f740 1 -- start start 2024-09-16T01:25:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.187+0000 7f82a6c0f740 1 Processor -- start 2024-09-16T01:25:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c6330 0x5636423e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1ab5b0 con 0x5636423c6330 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1aca20 con 0x5636423c5a10 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f829c1abc80 con 0x5636423d94c0 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.191+0000 7f82a6c0f740 10 client.0 _ll_register_callbacks cb 0x7f829400ff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.193+0000 7f82a2414640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.193+0000 7f82a2414640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.194+0000 7f82a2414640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:41812/0 (socket says 172.21.15.78:41812) 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.194+0000 7f82a2414640 1 -- 172.21.15.78:0/917535723 learned_addr learned my addr 172.21.15.78:0/917535723 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.196+0000 7f82a2414640 1 -- 172.21.15.78:0/917535723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 msgr2=0x5636423e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.196+0000 7f82a2414640 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423e1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.196+0000 7f82a2414640 1 -- 172.21.15.78:0/917535723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c6330 msgr2=0x5636423e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.196+0000 7f82a2414640 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c6330 0x5636423e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.196+0000 7f82a2414640 1 -- 172.21.15.78:0/917535723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636423da570 con 0x5636423c5a10 2024-09-16T01:25:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.201+0000 7f82a2414640 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8294004d40 tx=0x7f8294005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9548219d103b4cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.203+0000 7f82857fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:25:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.203+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294004e20 con 0x5636423c5a10 2024-09-16T01:25:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.204+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f82940052d0 con 0x5636423c5a10 2024-09-16T01:25:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.205+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f829400f9a0 con 0x5636423c5a10 2024-09-16T01:25:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.211+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f82940049d0 con 0x5636423c5a10 2024-09-16T01:25:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.216+0000 7f82a6c0f740 10 client.26288 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.216+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5636423ca840 con 0x5636423c5a10 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829400f9a0 con 0x5636423c5a10 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8294024040 con 0x5636423c5a10 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 10 client.26288 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 10 client.26288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 20 client.26288 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423caf10 0x5636423cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 4 client.26288 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:25:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.225+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563642331d70 con 0x5636423caf10 2024-09-16T01:25:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.232+0000 7f82a1412640 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423caf10 0x5636423cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.252+0000 7f82a1412640 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423caf10 0x5636423cd300 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27b1430bae8cefd6 server_cookie=803d278adceb9c9e in_seq=0 out_seq=0 2024-09-16T01:25:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.255+0000 7f82857fa640 10 client.26288 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:29.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:29 smithi078 ceph-mon[30937]: pgmap v462: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:25:29.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:29 smithi193 ceph-mon[35701]: pgmap v462: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:25:29.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.326+0000 7f82857fa640 1 -- 172.21.15.78:0/917535723 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563642331d70 con 0x5636423caf10 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.326+0000 7f82857fa640 10 client.26288 handle_command_reply: tid=0 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.326+0000 7f82a6c0f740 1 client.26288 shutdown 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.326+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 msgr2=0x563642451640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.326+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8294004d40 tx=0x7f8294005cf0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 shutdown_connections 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5636423caf10 0x5636423cd300 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5636423d94c0 0x5636423e1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636423c6330 0x5636423e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 --2- 172.21.15.78:0/917535723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636423c5a10 0x563642451640 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 >> 172.21.15.78:0/917535723 conn(0x5636423c2060 msgr2=0x5636423e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82857fa640 0 client.26288 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 shutdown_connections 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 1 -- 172.21.15.78:0/917535723 wait complete. 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.327+0000 7f82a6c0f740 20 client.26288 trim_cache size 0 max 16384 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:25:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:25:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 msgr2=0x7f82880a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 0x7f82880a5920 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f82980cd7f0 tx=0x7f8298025360 comp rx=0 tx=0).stop 2024-09-16T01:25:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 msgr2=0x7f829c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f829800e9f0 tx=0x7f829800eeb0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 shutdown_connections 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f82880a3460 0x7f82880a5920 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c103aa0 0x7f829c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f829c1008e0 0x7f829c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 --2- 172.21.15.78:0/1568346164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f829c0fff60 0x7f829c1984f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 >> 172.21.15.78:0/1568346164 conn(0x7f829c0770f0 msgr2=0x7f829c105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 shutdown_connections 2024-09-16T01:25:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:29.328+0000 7f82a3e77640 1 -- 172.21.15.78:0/1568346164 wait complete. 2024-09-16T01:25:29.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:29 smithi148 ceph-mon[35030]: pgmap v462: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-09-16T01:25:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 855.0 s, 579.8 tps, lat 178.143 ms stddev 231.860 2024-09-16T01:25:31.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:31 smithi078 ceph-mon[30937]: pgmap v463: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T01:25:31.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:31 smithi193 ceph-mon[35701]: pgmap v463: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T01:25:31.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:31 smithi148 ceph-mon[35030]: pgmap v463: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.19k op/s 2024-09-16T01:25:33.279 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:33 smithi078 ceph-mon[30937]: pgmap v464: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 9.5 MiB/s wr, 834 op/s 2024-09-16T01:25:33.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:33 smithi193 ceph-mon[35701]: pgmap v464: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 9.5 MiB/s wr, 834 op/s 2024-09-16T01:25:33.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:33 smithi148 ceph-mon[35030]: pgmap v464: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 9.5 MiB/s wr, 834 op/s 2024-09-16T01:25:35.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:35 smithi078 ceph-mon[30937]: pgmap v465: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-09-16T01:25:35.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:35 smithi193 ceph-mon[35701]: pgmap v465: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-09-16T01:25:35.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:35 smithi148 ceph-mon[35030]: pgmap v465: 129 pgs: 129 active+clean; 3.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-09-16T01:25:35.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 860.0 s, 586.2 tps, lat 170.847 ms stddev 212.162 2024-09-16T01:25:37.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:37 smithi078 ceph-mon[30937]: pgmap v466: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-09-16T01:25:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:37 smithi193 ceph-mon[35701]: pgmap v466: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-09-16T01:25:37.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:37 smithi148 ceph-mon[35030]: pgmap v466: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.2 MiB/s wr, 557 op/s 2024-09-16T01:25:39.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:39 smithi078 ceph-mon[30937]: pgmap v467: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 4.7 MiB/s wr, 497 op/s 2024-09-16T01:25:39.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:39 smithi193 ceph-mon[35701]: pgmap v467: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 4.7 MiB/s wr, 497 op/s 2024-09-16T01:25:39.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:39 smithi148 ceph-mon[35030]: pgmap v467: 129 pgs: 129 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 4.7 MiB/s wr, 497 op/s 2024-09-16T01:25:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 865.0 s, 877.8 tps, lat 114.998 ms stddev 172.217 2024-09-16T01:25:41.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:41 smithi193 ceph-mon[35701]: pgmap v468: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.0 MiB/s wr, 545 op/s 2024-09-16T01:25:41.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:41 smithi148 ceph-mon[35030]: pgmap v468: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.0 MiB/s wr, 545 op/s 2024-09-16T01:25:41.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:41 smithi078 ceph-mon[30937]: pgmap v468: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.0 MiB/s wr, 545 op/s 2024-09-16T01:25:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.488+0000 7f6580ea4640 1 -- 172.21.15.78:0/2414528909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1095d0 msgr2=0x7f657c109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.488+0000 7f6580ea4640 1 --2- 172.21.15.78:0/2414528909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1095d0 0x7f657c109a10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f6570009900 tx=0x7f657002a440 comp rx=0 tx=0).stop 2024-09-16T01:25:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.489+0000 7f6580ea4640 1 -- 172.21.15.78:0/2414528909 shutdown_connections 2024-09-16T01:25:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.489+0000 7f6580ea4640 1 --2- 172.21.15.78:0/2414528909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.489+0000 7f6580ea4640 1 --2- 172.21.15.78:0/2414528909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1095d0 0x7f657c109a10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.489+0000 7f6580ea4640 1 --2- 172.21.15.78:0/2414528909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1091f0 0x7f657c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.489+0000 7f6580ea4640 1 -- 172.21.15.78:0/2414528909 >> 172.21.15.78:0/2414528909 conn(0x7f657c06ce00 msgr2=0x7f657c06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.494+0000 7f6580ea4640 1 -- 172.21.15.78:0/2414528909 shutdown_connections 2024-09-16T01:25:42.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.499+0000 7f6580ea4640 1 -- 172.21.15.78:0/2414528909 wait complete. 2024-09-16T01:25:42.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.499+0000 7f6580ea4640 1 Processor -- start 2024-09-16T01:25:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.500+0000 7f6580ea4640 1 -- start start 2024-09-16T01:25:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1091f0 0x7f657c11a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 0x7f657c114910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f657c111370 con 0x7f657c1091f0 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1111f0 con 0x7f657c11b140 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f6580ea4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f657c1114f0 con 0x7f657c1095d0 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50800/0 (socket says 172.21.15.78:50800) 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f657a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1091f0 0x7f657c11a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.501+0000 7f657ad9d640 1 -- 172.21.15.78:0/1981071973 learned_addr learned my addr 172.21.15.78:0/1981071973 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f6579d9b640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 0x7f657c114910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f657ad9d640 1 -- 172.21.15.78:0/1981071973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 msgr2=0x7f657c114910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f657ad9d640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 0x7f657c114910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f657ad9d640 1 -- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1091f0 msgr2=0x7f657c11a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f657ad9d640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1091f0 0x7f657c11a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.503+0000 7f657ad9d640 1 -- 172.21.15.78:0/1981071973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c1155d0 con 0x7f657c11b140 2024-09-16T01:25:42.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.505+0000 7f6579d9b640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 0x7f657c114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:25:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.505+0000 7f657ad9d640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f657400cad0 tx=0x7f657400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.506+0000 7f656b7fe640 1 -- 172.21.15.78:0/1981071973 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657400ae20 con 0x7f657c11b140 2024-09-16T01:25:42.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.506+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c1bc050 con 0x7f657c11b140 2024-09-16T01:25:42.509 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.506+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c1bc510 con 0x7f657c11b140 2024-09-16T01:25:42.509 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.506+0000 7f656b7fe640 1 -- 172.21.15.78:0/1981071973 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6574005020 con 0x7f657c11b140 2024-09-16T01:25:42.509 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.506+0000 7f656b7fe640 1 -- 172.21.15.78:0/1981071973 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657400ae20 con 0x7f657c11b140 2024-09-16T01:25:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.510+0000 7f656b7fe640 1 -- 172.21.15.78:0/1981071973 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f657400f550 con 0x7f657c11b140 2024-09-16T01:25:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.511+0000 7f656b7fe640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 0x7f65640a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.515 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 Processor -- start 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 -- start start 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f656b7fe640 1 -- 172.21.15.78:0/1981071973 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f657400ac20 con 0x7f657c11b140 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c123020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f657a59c640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 0x7f65640a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557c122592f0 con 0x557c122f5410 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557c12201270 con 0x557c122ffcb0 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6583c37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557c122552b0 con 0x557c122f4af0 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.514+0000 7f6579d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.516 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.515+0000 7f6579d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45028/0 (socket says 172.21.15.78:45028) 2024-09-16T01:25:42.517 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.515+0000 7f6579d9b640 1 -- 172.21.15.78:0/680928379 learned_addr learned my addr 172.21.15.78:0/680928379 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:42.518 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.515+0000 7f657a59c640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.516+0000 7f657ad9d640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c123020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.519 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.518+0000 7f6579d9b640 1 -- 172.21.15.78:0/680928379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 msgr2=0x557c122f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.520+0000 7f6579d9b640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.520+0000 7f6579d9b640 1 -- 172.21.15.78:0/680928379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 msgr2=0x557c123020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.520+0000 7f6579d9b640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c123020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.520+0000 7f6579d9b640 1 -- 172.21.15.78:0/680928379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c12200340 con 0x557c122f5410 2024-09-16T01:25:42.521 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.520+0000 7f657a59c640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:25:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.523+0000 7f657ad9d640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c123020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:25:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.523+0000 7f657a59c640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 0x7f65640a5d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c009590 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.523+0000 7f6579d9b640 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6570006010 tx=0x7f6570004180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a07bcdd5c2009c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.524+0000 7f6568ff9640 1 -- 172.21.15.78:0/680928379 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657003f070 con 0x557c122f5410 2024-09-16T01:25:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.525+0000 7f6568ff9640 1 -- 172.21.15.78:0/680928379 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6570005520 con 0x557c122f5410 2024-09-16T01:25:42.526 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.525+0000 7f6568ff9640 1 -- 172.21.15.78:0/680928379 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570035030 con 0x557c122f5410 2024-09-16T01:25:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.526+0000 7f6583c37740 1 -- 172.21.15.78:0/680928379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 msgr2=0x557c122ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.526+0000 7f6583c37740 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6570006010 tx=0x7f6570004180 comp rx=0 tx=0).stop 2024-09-16T01:25:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.528+0000 7f6583c37740 1 -- 172.21.15.78:0/680928379 shutdown_connections 2024-09-16T01:25:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.528+0000 7f6583c37740 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c123020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.528+0000 7f6583c37740 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122ff770 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.528+0000 7f6583c37740 1 --2- 172.21.15.78:0/680928379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.528+0000 7f6583c37740 1 -- 172.21.15.78:0/680928379 >> 172.21.15.78:0/680928379 conn(0x557c122f1060 msgr2=0x557c122f1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.533+0000 7f6583c37740 1 -- 172.21.15.78:0/680928379 shutdown_connections 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.539+0000 7f6583c37740 1 -- 172.21.15.78:0/680928379 wait complete. 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.539+0000 7f6583c37740 1 -- start start 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.539+0000 7f6583c37740 1 Processor -- start 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122fd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557c12302880 con 0x557c122f5410 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557c121e2290 con 0x557c122ffcb0 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557c12302a00 con 0x557c122f4af0 2024-09-16T01:25:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6583c37740 10 client.0 _ll_register_callbacks cb 0x557c12213140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6579d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6579d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45030/0 (socket says 172.21.15.78:45030) 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.541+0000 7f6579d9b640 1 -- 172.21.15.78:0/1872513121 learned_addr learned my addr 172.21.15.78:0/1872513121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f657ad9d640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122fd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f6579d9b640 1 -- 172.21.15.78:0/1872513121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 msgr2=0x557c122fcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f6579d9b640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f6579d9b640 1 -- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 msgr2=0x557c122fd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f6579d9b640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122fd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.542+0000 7f6579d9b640 1 -- 172.21.15.78:0/1872513121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c11f6c210 con 0x557c122f5410 2024-09-16T01:25:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.543+0000 7f6579d9b640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6570000c00 tx=0x7f65700045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45580cf59a16ad31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.543+0000 7f6561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:25:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.543+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657004c070 con 0x557c122f5410 2024-09-16T01:25:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.543+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f6570004da0 con 0x557c122f5410 2024-09-16T01:25:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.544+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6570005520 con 0x557c122f5410 2024-09-16T01:25:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.547+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f657003f050 con 0x557c122f5410 2024-09-16T01:25:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.548+0000 7f6583c37740 10 client.17034 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.548+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557c12038620 con 0x557c122f5410 2024-09-16T01:25:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.548+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6570005520 con 0x557c122f5410 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6570059020 con 0x557c122f5410 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 10 client.17034 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 10 client.17034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 20 client.17034 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1230f9f0 0x557c12311de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 4 client.17034 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c12266a20 con 0x557c1230f9f0 2024-09-16T01:25:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.549+0000 7f657a59c640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1230f9f0 0x557c12311de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.550+0000 7f6561ffb640 10 client.17034 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.551+0000 7f657a59c640 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1230f9f0 0x557c12311de0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6bc1af2f46a257 server_cookie=71d1d19fc4ed353f in_seq=0 out_seq=0 2024-09-16T01:25:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.552+0000 7f6561ffb640 1 -- 172.21.15.78:0/1872513121 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557c12266a20 con 0x557c1230f9f0 2024-09-16T01:25:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.552+0000 7f6561ffb640 10 client.17034 handle_command_reply: tid=0 2024-09-16T01:25:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.552+0000 7f6583c37740 1 client.17034 shutdown 2024-09-16T01:25:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.552+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 msgr2=0x557c122fd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.552+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6570000c00 tx=0x7f65700045a0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 shutdown_connections 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1230f9f0 0x557c12311de0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122fd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c122fd090 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 --2- 172.21.15.78:0/1872513121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f4af0 0x557c122fcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 >> 172.21.15.78:0/1872513121 conn(0x557c122f1060 msgr2=0x557c122f01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6561ffb640 0 client.17034 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 shutdown_connections 2024-09-16T01:25:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 1 -- 172.21.15.78:0/1872513121 wait complete. 2024-09-16T01:25:42.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.554+0000 7f6583c37740 20 client.17034 trim_cache size 0 max 16384 2024-09-16T01:25:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.571+0000 7f6583c37740 1 Processor -- start 2024-09-16T01:25:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.571+0000 7f6583c37740 1 -- start start 2024-09-16T01:25:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f5410 0x557c122fd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f655c0129a0 con 0x557c122f4af0 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f655c010270 con 0x557c122ffcb0 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6583c37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f655c00e0e0 con 0x557c122f5410 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f657a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f657a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45034/0 (socket says 172.21.15.78:45034) 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:50832/0 (socket says 172.21.15.78:50832) 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f657a59c640 1 -- 172.21.15.78:0/3283841218 learned_addr learned my addr 172.21.15.78:0/3283841218 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.572+0000 7f6579d9b640 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f5410 0x557c122fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 -- 172.21.15.78:0/3283841218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f5410 msgr2=0x557c122fd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f5410 0x557c122fd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 -- 172.21.15.78:0/3283841218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 msgr2=0x557c122f7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 -- 172.21.15.78:0/3283841218 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c122ec2d0 con 0x557c122ffcb0 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657a59c640 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.573+0000 7f657ad9d640 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65740ccdf0 tx=0x7f65740a46c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653efc03ff2c9541 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.574+0000 7f6563fff640 1 -- 172.21.15.78:0/3283841218 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657401d040 con 0x557c122ffcb0 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.574+0000 7f6563fff640 1 -- 172.21.15.78:0/3283841218 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65740cd070 con 0x557c122ffcb0 2024-09-16T01:25:42.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.574+0000 7f6563fff640 1 -- 172.21.15.78:0/3283841218 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574097b60 con 0x557c122ffcb0 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- 172.21.15.78:0/3283841218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 msgr2=0x557c122feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65740ccdf0 tx=0x7f65740a46c0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- 172.21.15.78:0/3283841218 shutdown_connections 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122ffcb0 0x557c122feac0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122f5410 0x557c122fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 --2- 172.21.15.78:0/3283841218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f4af0 0x557c122f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- 172.21.15.78:0/3283841218 >> 172.21.15.78:0/3283841218 conn(0x557c122f1060 msgr2=0x557c122fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- 172.21.15.78:0/3283841218 shutdown_connections 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- 172.21.15.78:0/3283841218 wait complete. 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 -- start start 2024-09-16T01:25:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.575+0000 7f6583c37740 1 Processor -- start 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 0x557c12307eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c123083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f655c00e730 con 0x557c122f5410 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f655c00fba0 con 0x557c122f4af0 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f655c00ee00 con 0x557c122ffcb0 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6583c37740 10 client.0 _ll_register_callbacks cb 0x7f65700040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6579d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c123083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:52890/0 (socket says 172.21.15.78:52890) 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 -- 172.21.15.78:0/2786154696 learned_addr learned my addr 172.21.15.78:0/2786154696 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 0x557c12307eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 -- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 msgr2=0x557c12307eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 0x557c12307eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 -- 172.21.15.78:0/2786154696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 msgr2=0x557c123083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c123083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 -- 172.21.15.78:0/2786154696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c122f9b90 con 0x557c122ffcb0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657a59c640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 0x557c12307eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f657ad9d640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f65740a1010 tx=0x7f6574004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b7f2d668715407 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.576+0000 7f6561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65740b4060 con 0x557c122ffcb0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65740cd070 con 0x557c122ffcb0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f65740cd220 con 0x557c122ffcb0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f657401d040 con 0x557c122ffcb0 2024-09-16T01:25:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 10 client.26335 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557c1237a000 con 0x557c122ffcb0 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65740cd070 con 0x557c122ffcb0 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6574086030 con 0x557c122ffcb0 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 10 client.26335 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 10 client.26335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 20 client.26335 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1237a570 0x557c1237c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 4 client.26335 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557c12260d70 con 0x557c1237a570 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.577+0000 7f657a59c640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1237a570 0x557c1237c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.578+0000 7f6561ffb640 10 client.26335 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.578+0000 7f657a59c640 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1237a570 0x557c1237c990 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b71ea56e1d85e4d2 server_cookie=94de11a12b38dfc8 in_seq=0 out_seq=0 2024-09-16T01:25:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.584+0000 7f6561ffb640 1 -- 172.21.15.78:0/2786154696 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557c12260d70 con 0x557c1237a570 2024-09-16T01:25:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.584+0000 7f6561ffb640 10 client.26335 handle_command_reply: tid=0 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 client.26335 shutdown 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 msgr2=0x557c122f9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f65740a1010 tx=0x7f6574004d30 comp rx=0 tx=0).stop 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 shutdown_connections 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557c1237a570 0x557c1237c990 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c122ffcb0 0x557c122f9360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6561ffb640 0 client.26335 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c122f5410 0x557c123083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 --2- 172.21.15.78:0/2786154696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557c122f4af0 0x557c12307eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 >> 172.21.15.78:0/2786154696 conn(0x557c122f1060 msgr2=0x557c122f01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 shutdown_connections 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 1 -- 172.21.15.78:0/2786154696 wait complete. 2024-09-16T01:25:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.585+0000 7f6583c37740 20 client.26335 trim_cache size 0 max 16384 2024-09-16T01:25:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:25:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:25:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:25:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 msgr2=0x7f65640a5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 0x7f65640a5d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c009590 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 msgr2=0x7f657c114e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f657400cad0 tx=0x7f657400cf90 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 shutdown_connections 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f65640a38c0 0x7f65640a5d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657c11b140 0x7f657c114e50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f657c1095d0 0x7f657c114910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 --2- 172.21.15.78:0/1981071973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f657c1091f0 0x7f657c11a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 >> 172.21.15.78:0/1981071973 conn(0x7f657c06ce00 msgr2=0x7f657c070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:42.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.586+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 shutdown_connections 2024-09-16T01:25:42.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:42.587+0000 7f6580ea4640 1 -- 172.21.15.78:0/1981071973 wait complete. 2024-09-16T01:25:42.831 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-16T01:25:42.832 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-16T01:25:42.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:25:43.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:43 smithi078 ceph-mon[30937]: pgmap v469: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 546 op/s 2024-09-16T01:25:43.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:43 smithi193 ceph-mon[35701]: pgmap v469: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 546 op/s 2024-09-16T01:25:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:25:43.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:43 smithi148 ceph-mon[35030]: pgmap v469: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.1 MiB/s wr, 546 op/s 2024-09-16T01:25:45.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:45 smithi078 ceph-mon[30937]: pgmap v470: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-09-16T01:25:45.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:45 smithi193 ceph-mon[35701]: pgmap v470: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-09-16T01:25:45.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:45 smithi148 ceph-mon[35030]: pgmap v470: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 MiB/s wr, 486 op/s 2024-09-16T01:25:45.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 870.0 s, 651.8 tps, lat 153.910 ms stddev 187.824 2024-09-16T01:25:47.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:47 smithi148 ceph-mon[35030]: pgmap v471: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 572 op/s 2024-09-16T01:25:47.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:47 smithi078 ceph-mon[30937]: pgmap v471: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 572 op/s 2024-09-16T01:25:47.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:47 smithi193 ceph-mon[35701]: pgmap v471: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 572 op/s 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/3298582710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd01008c0 msgr2=0x7f8bd0100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/3298582710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd01008c0 0x7f8bd0100ca0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009860 tx=0x7f8bc002a340 comp rx=0 tx=0).stop 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/3298582710 shutdown_connections 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/3298582710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd0105ec0 0x7f8bd0106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/3298582710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01011e0 0x7f8bd0105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/3298582710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd01008c0 0x7f8bd0100ca0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/3298582710 >> 172.21.15.78:0/3298582710 conn(0x7f8bd00fc7d0 msgr2=0x7f8bd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/3298582710 shutdown_connections 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.824+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/3298582710 wait complete. 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.825+0000 7f8bd4cdf640 1 Processor -- start 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.825+0000 7f8bd4cdf640 1 -- start start 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01008c0 0x7f8bd01a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd0105ec0 0x7f8bd019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0118630 con 0x7f8bd0105ec0 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01184b0 con 0x7f8bd01011e0 2024-09-16T01:25:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bd4cdf640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd01187b0 con 0x7f8bd01008c0 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bced9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd0105ec0 0x7f8bd019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:39954/0 (socket says 172.21.15.78:39954) 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 -- 172.21.15.78:0/2542502151 learned_addr learned my addr 172.21.15.78:0/2542502151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bce59c640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01008c0 0x7f8bd01a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 -- 172.21.15.78:0/2542502151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01008c0 msgr2=0x7f8bd01a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01008c0 0x7f8bd01a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 -- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd0105ec0 msgr2=0x7f8bd019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd0105ec0 0x7f8bd019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 -- 172.21.15.78:0/2542502151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd019f130 con 0x7f8bd01011e0 2024-09-16T01:25:48.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.826+0000 7f8bcdd9b640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e9f0 tx=0x7f8bc400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.827+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc400cd80 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.827+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4005500 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.827+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4010690 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.827+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd019f420 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.827+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd01ab710 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.828+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd006a850 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.829+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f8bc40046c0 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.830+0000 7f8bbf7fe640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 0x7f8bb80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.830+0000 7f8bce59c640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 0x7f8bb80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.830+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f8bc40d0730 con 0x7f8bd01011e0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.831+0000 7f8bce59c640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 0x7f8bb80a58f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009830 tx=0x7f8bc00058d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:48.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:47.833+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8bc409ccc0 con 0x7f8bd01011e0 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.207+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bd0105980 con 0x7f8bd01011e0 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.208+0000 7f8bbf7fe640 1 -- 172.21.15.78:0/2542502151 <== mon.2 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f8bd0105980 con 0x7f8bd01011e0 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.212+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 msgr2=0x7f8bb80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.212+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 0x7f8bb80a58f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009830 tx=0x7f8bc00058d0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.212+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 msgr2=0x7f8bd01a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.212+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400e9f0 tx=0x7f8bc400edf0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 shutdown_connections 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f8bb80a3430 0x7f8bb80a58f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bd0105ec0 0x7f8bd019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bd01011e0 0x7f8bd01a4d80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 --2- 172.21.15.78:0/2542502151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8bd01008c0 0x7f8bd01a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.215+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 >> 172.21.15.78:0/2542502151 conn(0x7f8bd00fc7d0 msgr2=0x7f8bd0103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.216+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 shutdown_connections 2024-09-16T01:25:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:48.216+0000 7f8bd4cdf640 1 -- 172.21.15.78:0/2542502151 wait complete. 2024-09-16T01:25:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:49 smithi148 ceph-mon[35030]: pgmap v472: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 MiB/s wr, 504 op/s 2024-09-16T01:25:49.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:49 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2542502151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:25:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:49 smithi078 ceph-mon[30937]: pgmap v472: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 MiB/s wr, 504 op/s 2024-09-16T01:25:49.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:49 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2542502151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:25:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:49 smithi193 ceph-mon[35701]: pgmap v472: 129 pgs: 129 active+clean; 3.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 MiB/s wr, 504 op/s 2024-09-16T01:25:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:49 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2542502151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:25:50.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 875.0 s, 586.4 tps, lat 168.755 ms stddev 218.673 2024-09-16T01:25:51.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:51 smithi148 ceph-mon[35030]: pgmap v473: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-16T01:25:51.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:51 smithi078 ceph-mon[30937]: pgmap v473: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-16T01:25:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:51 smithi193 ceph-mon[35701]: pgmap v473: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-16T01:25:51.810 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-16T01:25:51.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-16T01:25:52.409 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 -- 172.21.15.78:0/470498325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b41091f0 msgr2=0x7f45b41095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 --2- 172.21.15.78:0/470498325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b41091f0 0x7f45b41095d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009900 tx=0x7f45a402a440 comp rx=0 tx=0).stop 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 -- 172.21.15.78:0/470498325 shutdown_connections 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 --2- 172.21.15.78:0/470498325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b4119290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 --2- 172.21.15.78:0/470498325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45b4109b10 0x7f45b4116960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 --2- 172.21.15.78:0/470498325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b41091f0 0x7f45b41095d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.031+0000 7f45bb088640 1 -- 172.21.15.78:0/470498325 >> 172.21.15.78:0/470498325 conn(0x7f45b406e140 msgr2=0x7f45b406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.032+0000 7f45bb088640 1 -- 172.21.15.78:0/470498325 shutdown_connections 2024-09-16T01:25:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.032+0000 7f45bb088640 1 -- 172.21.15.78:0/470498325 wait complete. 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 Processor -- start 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 -- start start 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45b41091f0 0x7f45b410c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b410d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f45b411ca60 con 0x7f45b4116ea0 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f45b411c8e0 con 0x7f45b41091f0 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45bb088640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45b411cbe0 con 0x7f45b4109b10 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.033+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35312/0 (socket says 172.21.15.78:35312) 2024-09-16T01:25:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 -- 172.21.15.78:0/353499869 learned_addr learned my addr 172.21.15.78:0/353499869 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b9625640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b410d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 -- 172.21.15.78:0/353499869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45b41091f0 msgr2=0x7f45b410c5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45b41091f0 0x7f45b410c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 -- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 msgr2=0x7f45b410d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b410d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 -- 172.21.15.78:0/353499869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b41a2cc0 con 0x7f45b4109b10 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b9625640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b410d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b3fff640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f45a800b480 tx=0x7f45a800b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b1ffb640 1 -- 172.21.15.78:0/353499869 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8014070 con 0x7f45b4109b10 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b1ffb640 1 -- 172.21.15.78:0/353499869 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a8010d50 con 0x7f45b4109b10 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45b1ffb640 1 -- 172.21.15.78:0/353499869 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8017360 con 0x7f45b4109b10 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.034+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b41a2fb0 con 0x7f45b4109b10 2024-09-16T01:25:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.035+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b41a3440 con 0x7f45b4109b10 2024-09-16T01:25:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 Processor -- start 2024-09-16T01:25:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 -- start start 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73ba5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73ba7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562b73b002a0 con 0x562b73b9b080 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562b73aa7b10 con 0x562b73ba5940 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.038+0000 7f45bd461740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b73afc260 con 0x562b73b9a760 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35318/0 (socket says 172.21.15.78:35318) 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 -- 172.21.15.78:0/3237419872 learned_addr learned my addr 172.21.15.78:0/3237419872 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b8e24640 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73ba5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 -- 172.21.15.78:0/3237419872 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 msgr2=0x562b73ba7d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:25:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73ba7d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 -- 172.21.15.78:0/3237419872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 msgr2=0x562b73ba5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73ba5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 -- 172.21.15.78:0/3237419872 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b73871b90 con 0x562b73b9a760 2024-09-16T01:25:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.039+0000 7f45b9625640 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00f410 tx=0x7f45ac00f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcacb744ebb91e43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.040+0000 7f459affd640 1 -- 172.21.15.78:0/3237419872 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011700 con 0x562b73b9a760 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.040+0000 7f459affd640 1 -- 172.21.15.78:0/3237419872 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45ac011d80 con 0x562b73b9a760 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.040+0000 7f459affd640 1 -- 172.21.15.78:0/3237419872 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac01b410 con 0x562b73b9a760 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45b1ffb640 1 -- 172.21.15.78:0/353499869 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f45a8017070 con 0x7f45b4109b10 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 -- 172.21.15.78:0/3237419872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 msgr2=0x562b73b9ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00f410 tx=0x7f45ac00f8d0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 -- 172.21.15.78:0/3237419872 shutdown_connections 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73ba7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73ba5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 --2- 172.21.15.78:0/3237419872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73b9ab40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 -- 172.21.15.78:0/3237419872 >> 172.21.15.78:0/3237419872 conn(0x562b73b96cf0 msgr2=0x562b73b970d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 -- 172.21.15.78:0/3237419872 shutdown_connections 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.041+0000 7f45bd461740 1 -- 172.21.15.78:0/3237419872 wait complete. 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45b1ffb640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 0x7f459c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 -- start start 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 Processor -- start 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45b1ffb640 1 -- 172.21.15.78:0/353499869 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f45a80cf920 con 0x7f45b4109b10 2024-09-16T01:25:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45b9625640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 0x7f459c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73ba4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73ba4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562b73ba8510 con 0x562b73ba5940 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562b73a886e0 con 0x562b73b9a760 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b73ba8690 con 0x562b73b9b080 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.042+0000 7f45bd461740 10 client.0 _ll_register_callbacks cb 0x562b73aaca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:39972/0 (socket says 172.21.15.78:39972) 2024-09-16T01:25:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 -- 172.21.15.78:0/2044302623 learned_addr learned my addr 172.21.15.78:0/2044302623 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b3fff640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73ba4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 -- 172.21.15.78:0/2044302623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 msgr2=0x562b73ba4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73ba4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 -- 172.21.15.78:0/2044302623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 msgr2=0x562b73ba4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73ba4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.043+0000 7f45b8e24640 1 -- 172.21.15.78:0/2044302623 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b73bb1a30 con 0x562b73b9a760 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.044+0000 7f45b8e24640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45a4000c00 tx=0x7f45a40045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6abcdfd58155e68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.044+0000 7f4593fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:25:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4040070 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f45a4005550 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45ac011d80 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45a404c050 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f45bd461740 10 client.26306 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b73bb4410 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a4004ba0 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.045+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f45a403a040 con 0x562b73b9a760 2024-09-16T01:25:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 10 client.26306 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 10 client.26306 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 20 client.26306 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73bb7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 4 client.26306 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f45ac011d80 con 0x562b73bb58d0 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.046+0000 7f45b9625640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 0x7f459c0a5780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45ac010a50 tx=0x7f45ac005fd0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.047+0000 7f45b3fff640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73bb7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.047+0000 7f45b3fff640 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73bb7cc0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30465e2c7688b32e server_cookie=612b262dab03fd91 in_seq=0 out_seq=0 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.047+0000 7f4593fff640 10 client.26306 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f4593fff640 1 -- 172.21.15.78:0/2044302623 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f45ac011d80 con 0x562b73bb58d0 2024-09-16T01:25:53.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f4593fff640 10 client.26306 handle_command_reply: tid=0 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 client.26306 shutdown 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 msgr2=0x562b73ba40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45a4000c00 tx=0x7f45a40045d0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 shutdown_connections 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73bb7cc0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73ba4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f4593fff640 0 client.26306 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73ba4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.048+0000 7f45bd461740 1 --2- 172.21.15.78:0/2044302623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73ba40f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.049+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 >> 172.21.15.78:0/2044302623 conn(0x562b73b96cf0 msgr2=0x562b73b9ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.049+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 shutdown_connections 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.049+0000 7f45bd461740 1 -- 172.21.15.78:0/2044302623 wait complete. 2024-09-16T01:25:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.049+0000 7f45bd461740 20 client.26306 trim_cache size 0 max 16384 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 Processor -- start 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 -- start start 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73ba3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73bbd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73bbda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41c9200 con 0x562b73b9b080 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41c6ad0 con 0x562b73ba5940 2024-09-16T01:25:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.071+0000 7f45bd461740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41095d0 con 0x562b73b9a760 2024-09-16T01:25:53.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73bbd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b8e24640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73ba3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73bbda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73bbd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48102/0 (socket says 172.21.15.78:48102) 2024-09-16T01:25:53.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b3fff640 1 -- 172.21.15.78:0/1154470115 learned_addr learned my addr 172.21.15.78:0/1154470115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:53.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 -- 172.21.15.78:0/1154470115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 msgr2=0x562b73ba3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73ba3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 -- 172.21.15.78:0/1154470115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 msgr2=0x562b73bbd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73bbd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 -- 172.21.15.78:0/1154470115 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b73b92260 con 0x562b73ba5940 2024-09-16T01:25:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.072+0000 7f45b9625640 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73bbda80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00fdd0 tx=0x7f45ac009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e416f33f2d1f494f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.077+0000 7f459a7fc640 1 -- 172.21.15.78:0/1154470115 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011500 con 0x562b73ba5940 2024-09-16T01:25:53.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.077+0000 7f459a7fc640 1 -- 172.21.15.78:0/1154470115 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45ac02d030 con 0x562b73ba5940 2024-09-16T01:25:53.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.077+0000 7f459a7fc640 1 -- 172.21.15.78:0/1154470115 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac005640 con 0x562b73ba5940 2024-09-16T01:25:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- 172.21.15.78:0/1154470115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 msgr2=0x562b73bbda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73bbda80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45ac00fdd0 tx=0x7f45ac009500 comp rx=0 tx=0).stop 2024-09-16T01:25:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- 172.21.15.78:0/1154470115 shutdown_connections 2024-09-16T01:25:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73ba5940 0x562b73bbda80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73b9b080 0x562b73bbd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 --2- 172.21.15.78:0/1154470115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9a760 0x562b73ba3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- 172.21.15.78:0/1154470115 >> 172.21.15.78:0/1154470115 conn(0x562b73b96cf0 msgr2=0x562b73ba4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- 172.21.15.78:0/1154470115 shutdown_connections 2024-09-16T01:25:53.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- 172.21.15.78:0/1154470115 wait complete. 2024-09-16T01:25:53.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.078+0000 7f45bd461740 1 -- start start 2024-09-16T01:25:53.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 Processor -- start 2024-09-16T01:25:53.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73bad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73baa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41c4f90 con 0x562b73ba5940 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41c6400 con 0x562b73b9a760 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45b41c5660 con 0x562b73b9b080 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45b9625640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73baa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45b3fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:35338/0 (socket says 172.21.15.78:35338) 2024-09-16T01:25:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45b3fff640 1 -- 172.21.15.78:0/4026601062 learned_addr learned my addr 172.21.15.78:0/4026601062 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45bd461740 10 client.0 _ll_register_callbacks cb 0x7f45a4004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.079+0000 7f45b8e24640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73bad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 -- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 msgr2=0x562b73bad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73bad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 -- 172.21.15.78:0/4026601062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 msgr2=0x562b73baa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:53.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73baa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 -- 172.21.15.78:0/4026601062 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b73baadf0 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45b3fff640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f45a809b8e0 tx=0x7f45a80a2040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32fcbaa39090555c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f4593fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80b3020 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f45a8097fc0 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45ac02d030 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45a801e040 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45bd461740 10 client.26359 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b73c20be0 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.080+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45ac02d030 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f45a80cf450 con 0x562b73b9b080 2024-09-16T01:25:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 10 client.26359 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 10 client.26359 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 20 client.26359 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73c233c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 4 client.26359 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x562b73b0d9d0 con 0x562b73bb58d0 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45b8e24640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73c233c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f4593fff640 10 client.26359 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.081+0000 7f45b8e24640 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73c233c0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88a42a70012dec14 server_cookie=332a5e7b789939fb in_seq=0 out_seq=0 2024-09-16T01:25:53.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:53 smithi078 ceph-mon[30937]: pgmap v474: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 MiB/s wr, 757 op/s 2024-09-16T01:25:53.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:53 smithi148 ceph-mon[35030]: pgmap v474: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 MiB/s wr, 757 op/s 2024-09-16T01:25:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f4593fff640 1 -- 172.21.15.78:0/4026601062 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562b73b0d9d0 con 0x562b73bb58d0 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f4593fff640 10 client.26359 handle_command_reply: tid=0 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 client.26359 shutdown 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 msgr2=0x562b73bade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f45a809b8e0 tx=0x7f45a80a2040 comp rx=0 tx=0).stop 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 shutdown_connections 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562b73bb58d0 0x562b73c233c0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562b73ba5940 0x562b73baa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b73b9b080 0x562b73bade20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f4593fff640 0 client.26359 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 --2- 172.21.15.78:0/4026601062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b73b9a760 0x562b73bad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 >> 172.21.15.78:0/4026601062 conn(0x562b73b96cf0 msgr2=0x562b73b95650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 shutdown_connections 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 1 -- 172.21.15.78:0/4026601062 wait complete. 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.541+0000 7f45bd461740 20 client.26359 trim_cache size 0 max 16384 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 54, 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "87499b9d": { 2024-09-16T01:25:53.544 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 54, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 137.7, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:25:53.545 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:25:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 msgr2=0x7f459c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 0x7f459c0a5780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45ac010a50 tx=0x7f45ac005fd0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 msgr2=0x7f45b410cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f45a800b480 tx=0x7f45a800b940 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 shutdown_connections 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f459c0a32c0 0x7f459c0a5780 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b4116ea0 0x7f45b410d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4109b10 0x7f45b410cb20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 --2- 172.21.15.78:0/353499869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45b41091f0 0x7f45b410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 >> 172.21.15.78:0/353499869 conn(0x7f45b406e140 msgr2=0x7f45b41173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 shutdown_connections 2024-09-16T01:25:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:25:53.542+0000 7f45bb088640 1 -- 172.21.15.78:0/353499869 wait complete. 2024-09-16T01:25:53.548 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:53 smithi193 ceph-mon[35701]: pgmap v474: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 7.7 MiB/s wr, 757 op/s 2024-09-16T01:25:54.331 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '87499b9d', quiesce took 11.5 seconds. Will release after: 6.199999999999999 2024-09-16T01:25:55.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:55 smithi148 ceph-mon[35030]: pgmap v475: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.5 MiB/s wr, 723 op/s 2024-09-16T01:25:55.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:55 smithi078 ceph-mon[30937]: pgmap v475: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.5 MiB/s wr, 723 op/s 2024-09-16T01:25:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:55 smithi193 ceph-mon[35701]: pgmap v475: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.5 MiB/s wr, 723 op/s 2024-09-16T01:25:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3472626870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34107b70 msgr2=0x7f3e34107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3472626870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34107b70 0x7f3e34107f50 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009cd0 tx=0x7f3e2402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3472626870 shutdown_connections 2024-09-16T01:25:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3472626870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34102470 0x7f3e3410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3472626870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 0x7f3e34101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3472626870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34107b70 0x7f3e34107f50 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3472626870 >> 172.21.15.78:0/3472626870 conn(0x7f3e340fd7d0 msgr2=0x7f3e340ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3472626870 shutdown_connections 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3472626870 wait complete. 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 Processor -- start 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.684+0000 7f3e3c4a4640 1 -- start start 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 0x7f3e341982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 0x7f3e3419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34111f20 con 0x7f3e34102470 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e34111da0 con 0x7f3e34107b70 2024-09-16T01:25:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3c4a4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341120a0 con 0x7f3e34101ad0 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52934/0 (socket says 172.21.15.78:52934) 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3a240640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 0x7f3e341982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3083459746 learned_addr learned my addr 172.21.15.78:0/3083459746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 0x7f3e3419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3083459746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 msgr2=0x7f3e341982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 0x7f3e341982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3083459746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 msgr2=0x7f3e3419cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 0x7f3e3419cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3083459746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3419d2c0 con 0x7f3e34102470 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 0x7f3e3419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.685+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c920 tx=0x7f3e2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.686+0000 7f3e337fe640 1 -- 172.21.15.78:0/3083459746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007c70 con 0x7f3e34102470 2024-09-16T01:25:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.686+0000 7f3e337fe640 1 -- 172.21.15.78:0/3083459746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e28007e10 con 0x7f3e34102470 2024-09-16T01:25:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.686+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3419d5b0 con 0x7f3e34102470 2024-09-16T01:25:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.686+0000 7f3e337fe640 1 -- 172.21.15.78:0/3083459746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28018890 con 0x7f3e34102470 2024-09-16T01:25:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.686+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e341a4e40 con 0x7f3e34102470 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.688+0000 7f3e337fe640 1 -- 172.21.15.78:0/3083459746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3e28018a70 con 0x7f3e34102470 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 Processor -- start 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 -- start start 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde3af0 0x55852bde3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 0x55852bdf10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55852bd482f0 con 0x55852bde4410 2024-09-16T01:25:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55852bcf0270 con 0x55852bdeecb0 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3f236740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55852bd442b0 con 0x55852bde3af0 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52948/0 (socket says 172.21.15.78:52948) 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 -- 172.21.15.78:0/756341410 learned_addr learned my addr 172.21.15.78:0/756341410 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e337fe640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 0x7f3e1c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde3af0 0x55852bde3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 0x55852bdf10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 -- 172.21.15.78:0/756341410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde3af0 msgr2=0x55852bde3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde3af0 0x55852bde3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 -- 172.21.15.78:0/756341410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 msgr2=0x55852bdf10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 0x55852bdf10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3a240640 1 -- 172.21.15.78:0/756341410 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852bcef340 con 0x55852bde4410 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 0x55852bdf10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 0x7f3e1c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.689+0000 7f3e337fe640 1 -- 172.21.15.78:0/3083459746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f3e280d0260 con 0x7f3e34102470 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e3a240640 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009cd0 tx=0x7f3e240314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a24319913da0ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e30bfe640 1 -- 172.21.15.78:0/756341410 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24048070 con 0x55852bde4410 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e30bfe640 1 -- 172.21.15.78:0/756341410 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24036070 con 0x55852bde4410 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e30bfe640 1 -- 172.21.15.78:0/756341410 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004580 con 0x55852bde4410 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e3f236740 1 -- 172.21.15.78:0/756341410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 msgr2=0x55852bdee770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e3f236740 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009cd0 tx=0x7f3e240314e0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.690+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 0x7f3e1c0a5920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009c00 tx=0x7f3e2c0046c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- 172.21.15.78:0/756341410 shutdown_connections 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bdeecb0 0x55852bdf10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdee770 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- 172.21.15.78:0/756341410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde3af0 0x55852bde3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- 172.21.15.78:0/756341410 >> 172.21.15.78:0/756341410 conn(0x55852bde0060 msgr2=0x55852bde0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- 172.21.15.78:0/756341410 shutdown_connections 2024-09-16T01:25:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- 172.21.15.78:0/756341410 wait complete. 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- start start 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 Processor -- start 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdebb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852bdec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55852bdf1880 con 0x55852bde4410 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55852bcd1290 con 0x55852bde3af0 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55852bdf1a00 con 0x55852bdeecb0 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e3f236740 10 client.0 _ll_register_callbacks cb 0x55852bd02140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.691+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52962/0 (socket says 172.21.15.78:52962) 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 -- 172.21.15.78:0/1032059921 learned_addr learned my addr 172.21.15.78:0/1032059921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3a240640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdebb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852bdec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 -- 172.21.15.78:0/1032059921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 msgr2=0x55852bdec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852bdec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 -- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 msgr2=0x55852bdebb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdebb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 -- 172.21.15.78:0/1032059921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852ba5b210 con 0x55852bde4410 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852bdec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3e280cdbc0 tx=0x7f3e2809c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b010acff6d5b8225 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e280a0500 con 0x55852bde4410 2024-09-16T01:25:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f3e280bc0e0 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e24036070 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e280a4dc0 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3f236740 10 client.17073 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55852bb27620 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.692+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24036070 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3e280a4930 con 0x55852bde4410 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 10 client.17073 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 20 client.17073 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852bdfebe0 0x55852be00fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 4 client.17073 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55852bd55a20 con 0x55852bdfebe0 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3a240640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852bdfebe0 0x55852be00fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e3a240640 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852bdfebe0 0x55852be00fd0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f13345f8f65081 server_cookie=780e700d97850fcb in_seq=0 out_seq=0 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.693+0000 7f3e19ffb640 10 client.17073 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e19ffb640 1 -- 172.21.15.78:0/1032059921 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55852bd55a20 con 0x55852bdfebe0 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e19ffb640 10 client.17073 handle_command_reply: tid=0 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 client.17073 shutdown 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 msgr2=0x55852bdec0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3e280cdbc0 tx=0x7f3e2809c850 comp rx=0 tx=0).stop 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 shutdown_connections 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852bdfebe0 0x55852be00fd0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852bdec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852bdec0a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e19ffb640 0 client.17073 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 --2- 172.21.15.78:0/1032059921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdebb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 >> 172.21.15.78:0/1032059921 conn(0x55852bde0060 msgr2=0x55852bde5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 shutdown_connections 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.694+0000 7f3e3f236740 1 -- 172.21.15.78:0/1032059921 wait complete. 2024-09-16T01:25:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.695+0000 7f3e3f236740 20 client.17073 trim_cache size 0 max 16384 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.708+0000 7f3e3f236740 1 Processor -- start 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 -- start start 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852be06d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b1850 con 0x55852bde4410 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341af120 con 0x55852bde3af0 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e3f236740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341acf90 con 0x55852bdeecb0 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52964/0 (socket says 172.21.15.78:52964) 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3860543622 learned_addr learned my addr 172.21.15.78:0/3860543622 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:55.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3860543622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 msgr2=0x55852be06d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852be06d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3860543622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 msgr2=0x55852bdec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 -- 172.21.15.78:0/3860543622 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852bdfd780 con 0x55852bde4410 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3e280a01e0 tx=0x7f3e2800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4888cac8f81f7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e1bfff640 1 -- 172.21.15.78:0/3860543622 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800fb30 con 0x55852bde4410 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e1bfff640 1 -- 172.21.15.78:0/3860543622 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e280a4930 con 0x55852bde4410 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.709+0000 7f3e1bfff640 1 -- 172.21.15.78:0/3860543622 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2809b670 con 0x55852bde4410 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- 172.21.15.78:0/3860543622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 msgr2=0x55852be06840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3e280a01e0 tx=0x7f3e2800cf90 comp rx=0 tx=0).stop 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- 172.21.15.78:0/3860543622 shutdown_connections 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bdeecb0 0x55852be06d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bde4410 0x55852be06840 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 --2- 172.21.15.78:0/3860543622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- 172.21.15.78:0/3860543622 >> 172.21.15.78:0/3860543622 conn(0x55852bde0060 msgr2=0x55852bdedbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- 172.21.15.78:0/3860543622 shutdown_connections 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- 172.21.15.78:0/3860543622 wait complete. 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 -- start start 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.710+0000 7f3e3f236740 1 Processor -- start 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdf2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde4410 0x55852bdf3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341ad5e0 con 0x55852bdeecb0 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341aea50 con 0x55852bde3af0 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341adcb0 con 0x55852bde4410 2024-09-16T01:25:55.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 10 client.0 _ll_register_callbacks cb 0x7f3e280984c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52974/0 (socket says 172.21.15.78:52974) 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 -- 172.21.15.78:0/4032617541 learned_addr learned my addr 172.21.15.78:0/4032617541 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e39a3f640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde4410 0x55852bdf3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3a240640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdf2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 -- 172.21.15.78:0/4032617541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde4410 msgr2=0x55852bdf3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde4410 0x55852bdf3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 -- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 msgr2=0x55852bdf2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdf2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 -- 172.21.15.78:0/4032617541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852bde6940 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3aa41640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c01d6a0 tx=0x7f3e2c01daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638f6518a83070f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0126c0 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f3e2c012860 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.711+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e280a4930 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c012a60 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 10 client.17079 fetch_fsmap learned FSMap version 31 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55852be69c00 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e280a4930 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3e2c027420 con 0x55852bdeecb0 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 10 client.17079 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:25:55.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 10 client.17079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 20 client.17079 populate_metadata read hostname 'smithi078' 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852be6a3a0 0x55852be6c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 4 client.17079 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55852bd4fd70 con 0x55852be6a3a0 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.712+0000 7f3e3a240640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852be6a3a0 0x55852be6c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.713+0000 7f3e19ffb640 10 client.17079 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.713+0000 7f3e3a240640 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852be6a3a0 0x55852be6c7c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd76268cb9f85d9 server_cookie=2daed85ead4398f9 in_seq=0 out_seq=0 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.720+0000 7f3e19ffb640 1 -- 172.21.15.78:0/4032617541 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55852bd4fd70 con 0x55852be6a3a0 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.720+0000 7f3e19ffb640 10 client.17079 handle_command_reply: tid=0 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.720+0000 7f3e3f236740 1 client.17079 shutdown 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 msgr2=0x55852bdf36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c01d6a0 tx=0x7f3e2c01daa0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 shutdown_connections 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55852be6a3a0 0x55852be6c7c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55852bdeecb0 0x55852bdf36c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e19ffb640 0 client.17079 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55852bde4410 0x55852bdf3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 --2- 172.21.15.78:0/4032617541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55852bde3af0 0x55852bdf2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 >> 172.21.15.78:0/4032617541 conn(0x55852bde0060 msgr2=0x55852bdde9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 shutdown_connections 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 1 -- 172.21.15.78:0/4032617541 wait complete. 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3f236740 20 client.17079 trim_cache size 0 max 16384 2024-09-16T01:25:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:25:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:25:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:25:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:25:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 msgr2=0x7f3e1c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 0x7f3e1c0a5920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009c00 tx=0x7f3e2c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 msgr2=0x7f3e34198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c920 tx=0x7f3e2800cde0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 shutdown_connections 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3e1c0a3460 0x7f3e1c0a5920 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e34107b70 0x7f3e3419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e34102470 0x7f3e34198810 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 --2- 172.21.15.78:0/3083459746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e34101ad0 0x7f3e341982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 >> 172.21.15.78:0/3083459746 conn(0x7f3e340fd7d0 msgr2=0x7f3e340fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.721+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 shutdown_connections 2024-09-16T01:25:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:25:55.722+0000 7f3e3c4a4640 1 -- 172.21.15.78:0/3083459746 wait complete. 2024-09-16T01:25:55.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 880.0 s, 163.6 tps, lat 261.049 ms stddev 265.254 2024-09-16T01:25:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:57 smithi148 ceph-mon[35030]: pgmap v476: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 824 op/s 2024-09-16T01:25:57.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:57 smithi078 ceph-mon[30937]: pgmap v476: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 824 op/s 2024-09-16T01:25:57.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:57 smithi193 ceph-mon[35701]: pgmap v476: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 824 op/s 2024-09-16T01:25:59.349 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:25:59 smithi193 ceph-mon[35701]: pgmap v477: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.3 MiB/s wr, 719 op/s 2024-09-16T01:25:59.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:25:59 smithi148 ceph-mon[35030]: pgmap v477: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.3 MiB/s wr, 719 op/s 2024-09-16T01:25:59.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:25:59 smithi078 ceph-mon[30937]: pgmap v477: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.3 MiB/s wr, 719 op/s 2024-09-16T01:26:00.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:26:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:00.940 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 885.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:26:01.206 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:01 smithi078 ceph-mon[30937]: pgmap v478: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-09-16T01:26:01.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1260101363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 msgr2=0x7f3a58106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:01.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1260101363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a58106e60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009cd0 tx=0x7f3a5002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1260101363 shutdown_connections 2024-09-16T01:26:01.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1260101363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a5810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1260101363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 0x7f3a58100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1260101363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a58106e60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1260101363 >> 172.21.15.78:0/1260101363 conn(0x7f3a580fc7a0 msgr2=0x7f3a580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1260101363 shutdown_connections 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1260101363 wait complete. 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.283+0000 7f3a5f1a8640 1 Processor -- start 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 -- start start 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 0x7f3a58199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a58199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58110f20 con 0x7f3a58106a80 2024-09-16T01:26:01.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58110da0 con 0x7f3a58101420 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5f1a8640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a581110a0 con 0x7f3a58100a80 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52998/0 (socket says 172.21.15.78:52998) 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 -- 172.21.15.78:0/1882924149 learned_addr learned my addr 172.21.15.78:0/1882924149 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5cf44640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 0x7f3a58199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a57fff640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a58199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 -- 172.21.15.78:0/1882924149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 msgr2=0x7f3a58199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 0x7f3a58199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 -- 172.21.15.78:0/1882924149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 msgr2=0x7f3a58199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a58199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 -- 172.21.15.78:0/1882924149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a581a4a10 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a57fff640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a58199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.284+0000 7f3a5d745640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009fd0 tx=0x7f3a4800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.285+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48011830 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.285+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a48016070 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.285+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a581a4be0 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.285+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800f2e0 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.285+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a581a50f0 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.286+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a58100f60 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.287+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3a480119d0 con 0x7f3a58106a80 2024-09-16T01:26:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.287+0000 7f3a55ffb640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 0x7f3a440a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.288+0000 7f3a5cf44640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 0x7f3a440a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.288+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f3a480cfd90 con 0x7f3a58106a80 2024-09-16T01:26:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.288+0000 7f3a5cf44640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 0x7f3a440a58f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a50006010 tx=0x7f3a50009800 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.291+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3a4809c490 con 0x7f3a58106a80 2024-09-16T01:26:01.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:01 smithi148 ceph-mon[35030]: pgmap v478: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-09-16T01:26:01.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.491+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a5802d7c0 con 0x7f3a58106a80 2024-09-16T01:26:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.492+0000 7f3a55ffb640 1 -- 172.21.15.78:0/1882924149 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f3a4809bd90 con 0x7f3a58106a80 2024-09-16T01:26:01.494 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:01.494 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:26:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:26:01.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 msgr2=0x7f3a440a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 0x7f3a440a58f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a50006010 tx=0x7f3a50009800 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 msgr2=0x7f3a581a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009fd0 tx=0x7f3a4800b8e0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 shutdown_connections 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3a440a3430 0x7f3a440a58f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a58106a80 0x7f3a581a44d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3a58101420 0x7f3a58199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 --2- 172.21.15.78:0/1882924149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100a80 0x7f3a58199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 >> 172.21.15.78:0/1882924149 conn(0x7f3a580fc7a0 msgr2=0x7f3a580fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 shutdown_connections 2024-09-16T01:26:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:01.494+0000 7f3a5f1a8640 1 -- 172.21.15.78:0/1882924149 wait complete. 2024-09-16T01:26:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:01 smithi193 ceph-mon[35701]: pgmap v478: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-09-16T01:26:02.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:02 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1882924149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:02.486 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:02 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1882924149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:02.516 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 87499b9d --release --await' 2024-09-16T01:26:02.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 87499b9d --release --await 2024-09-16T01:26:02.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:02 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1882924149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:03.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:03 smithi078 ceph-mon[30937]: pgmap v479: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-09-16T01:26:03.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- 172.21.15.78:0/3271312829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050100a80 msgr2=0x7fa050100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- 172.21.15.78:0/3271312829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050100a80 0x7fa050100ee0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa044009930 tx=0x7fa04402a420 comp rx=0 tx=0).stop 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- 172.21.15.78:0/3271312829 shutdown_connections 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- 172.21.15.78:0/3271312829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 0x7fa05010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- 172.21.15.78:0/3271312829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050100a80 0x7fa050100ee0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- 172.21.15.78:0/3271312829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050106a80 0x7fa050106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- 172.21.15.78:0/3271312829 >> 172.21.15.78:0/3271312829 conn(0x7fa0500fc7a0 msgr2=0x7fa0500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- 172.21.15.78:0/3271312829 shutdown_connections 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- 172.21.15.78:0/3271312829 wait complete. 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 Processor -- start 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- start start 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 0x7fa0501983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 0x7fa050198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa050111f60 con 0x7fa050106a80 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa050111de0 con 0x7fa050101420 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.269+0000 7fa056781640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501120e0 con 0x7fa050100a80 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53010/0 (socket says 172.21.15.78:53010) 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 -- 172.21.15.78:0/3915315397 learned_addr learned my addr 172.21.15.78:0/3915315397 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 0x7fa050198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04ffff640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 0x7fa0501983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 -- 172.21.15.78:0/3915315397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 msgr2=0x7fa0501983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 0x7fa0501983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 -- 172.21.15.78:0/3915315397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 msgr2=0x7fa050198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 0x7fa050198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 -- 172.21.15.78:0/3915315397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05019d370 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04ffff640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 0x7fa0501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa054d1e640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eaa0 tx=0x7fa04800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04d7fa640 1 -- 172.21.15.78:0/3915315397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800cdb0 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05019d660 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04d7fa640 1 -- 172.21.15.78:0/3915315397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa048005020 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa04d7fa640 1 -- 172.21.15.78:0/3915315397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0480116b0 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.270+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0501a4f40 con 0x7fa050106a80 2024-09-16T01:26:03.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.272+0000 7fa04d7fa640 1 -- 172.21.15.78:0/3915315397 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fa048005560 con 0x7fa050106a80 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 Processor -- start 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 -- start start 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff72f9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7303d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557ff725e620 con 0x557ff7304240 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557ff7205e50 con 0x557ff72f9060 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa058b5a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557ff725a5e0 con 0x557ff72f9980 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53020/0 (socket says 172.21.15.78:53020) 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2421959229 learned_addr learned my addr 172.21.15.78:0/2421959229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04ffff640 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7303d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff72f9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04d7fa640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 0x7fa0380a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2421959229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 msgr2=0x557ff7303d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7303d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2421959229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 msgr2=0x557ff72f9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.273+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff72f9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2421959229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff71f6d40 con 0x557ff7304240 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa054d1e640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 0x7fa0380a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa044009930 tx=0x7fa0440056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf6f0a59288fdec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa04d7fa640 1 -- 172.21.15.78:0/3915315397 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7fa0480d0470 con 0x7fa050106a80 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa0367fc640 1 -- 172.21.15.78:0/2421959229 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044035040 con 0x557ff7304240 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa054d1e640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 0x7fa0380a5870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa048064070 tx=0x7fa048098680 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa0367fc640 1 -- 172.21.15.78:0/2421959229 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa044004030 con 0x557ff7304240 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.274+0000 7fa0367fc640 1 -- 172.21.15.78:0/2421959229 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044003be0 con 0x557ff7304240 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- 172.21.15.78:0/2421959229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 msgr2=0x557ff7306630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa044009930 tx=0x7fa0440056d0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- 172.21.15.78:0/2421959229 shutdown_connections 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7306630 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7303d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2421959229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff72f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- 172.21.15.78:0/2421959229 >> 172.21.15.78:0/2421959229 conn(0x557ff72f55f0 msgr2=0x557ff72f59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- 172.21.15.78:0/2421959229 shutdown_connections 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- 172.21.15.78:0/2421959229 wait complete. 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- start start 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 Processor -- start 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff7301130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7301670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557ff7306e10 con 0x557ff7304240 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557ff71e6f10 con 0x557ff72f9060 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557ff7306f90 con 0x557ff72f9980 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.275+0000 7fa058b5a740 10 client.0 _ll_register_callbacks cb 0x557ff720adb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53026/0 (socket says 172.21.15.78:53026) 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2133105718 learned_addr learned my addr 172.21.15.78:0/2133105718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04ffff640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7301670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff7301130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2133105718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 msgr2=0x557ff7301670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7301670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 msgr2=0x557ff7301130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff7301130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 -- 172.21.15.78:0/2133105718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff6f6d580 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa0440037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357c9590368ce76f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa02f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044035070 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7fa044044040 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa044004030 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa04405c050 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa058b5a740 10 client.17097 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.276+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ff7037960 con 0x557ff7304240 2024-09-16T01:26:03.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa044004030 con 0x557ff7304240 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa04403a040 con 0x557ff7304240 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 10 client.17097 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 10 client.17097 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 20 client.17097 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff7313c40 0x557ff7316030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 4 client.17097 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ff726bb90 con 0x557ff7313c40 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.277+0000 7fa04ffff640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff7313c40 0x557ff7316030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa04ffff640 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff7313c40 0x557ff7316030 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51d0bef1f80e6279 server_cookie=9014465a85df04fe in_seq=0 out_seq=0 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa02f7fe640 10 client.17097 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa02f7fe640 1 -- 172.21.15.78:0/2133105718 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557ff726bb90 con 0x557ff7313c40 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa02f7fe640 10 client.17097 handle_command_reply: tid=0 2024-09-16T01:26:03.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 client.17097 shutdown 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 msgr2=0x557ff7301bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa0440037c0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 shutdown_connections 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff7313c40 0x557ff7316030 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff7304240 0x557ff7301bb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa02f7fe640 0 client.17097 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff7301670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 --2- 172.21.15.78:0/2133105718 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff72f9060 0x557ff7301130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 >> 172.21.15.78:0/2133105718 conn(0x557ff72f55f0 msgr2=0x557ff72fb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.278+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 shutdown_connections 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.279+0000 7fa058b5a740 1 -- 172.21.15.78:0/2133105718 wait complete. 2024-09-16T01:26:03.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.279+0000 7fa058b5a740 20 client.17097 trim_cache size 0 max 16384 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.295+0000 7fa058b5a740 1 Processor -- start 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.295+0000 7fa058b5a740 1 -- start start 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.295+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9060 0x557ff7301800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.295+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.295+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff731bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa058b5a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501b18b0 con 0x557ff72f9980 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa058b5a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501af180 con 0x557ff7304240 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa058b5a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501acff0 con 0x557ff72f9060 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53032/0 (socket says 172.21.15.78:53032) 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 -- 172.21.15.78:0/77782796 learned_addr learned my addr 172.21.15.78:0/77782796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 -- 172.21.15.78:0/77782796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9060 msgr2=0x557ff7301800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9060 0x557ff7301800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 -- 172.21.15.78:0/77782796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 msgr2=0x557ff731bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff731bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 -- 172.21.15.78:0/77782796 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff73163a0 con 0x557ff72f9980 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa04ffff640 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa040004700 tx=0x7fa04000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2b03ccd362c5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa035ffb640 1 -- 172.21.15.78:0/77782796 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040005220 con 0x557ff72f9980 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa035ffb640 1 -- 172.21.15.78:0/77782796 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa040004c80 con 0x557ff72f9980 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.296+0000 7fa035ffb640 1 -- 172.21.15.78:0/77782796 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04001b980 con 0x557ff72f9980 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- 172.21.15.78:0/77782796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 msgr2=0x557ff731b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa040004700 tx=0x7fa04000f7f0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- 172.21.15.78:0/77782796 shutdown_connections 2024-09-16T01:26:03.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff731bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9980 0x557ff731b5d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 --2- 172.21.15.78:0/77782796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9060 0x557ff7301800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- 172.21.15.78:0/77782796 >> 172.21.15.78:0/77782796 conn(0x557ff72f55f0 msgr2=0x557ff73010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- 172.21.15.78:0/77782796 shutdown_connections 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- 172.21.15.78:0/77782796 wait complete. 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 -- start start 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.297+0000 7fa058b5a740 1 Processor -- start 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff730c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff73091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501ad640 con 0x557ff72f9060 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501aeab0 con 0x557ff7304240 2024-09-16T01:26:03.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0501add10 con 0x557ff72f9980 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 10 client.0 _ll_register_callbacks cb 0x7fa044004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53040/0 (socket says 172.21.15.78:53040) 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 -- 172.21.15.78:0/1787485176 learned_addr learned my addr 172.21.15.78:0/1787485176 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04ffff640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff730c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 -- 172.21.15.78:0/1787485176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 msgr2=0x557ff730c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff730c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 -- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 msgr2=0x557ff73091b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:03.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff73091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 -- 172.21.15.78:0/1787485176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff73096f0 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04ffff640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff730c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa04f7fe640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa044031bf0 tx=0x7fa044009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43556d692dafa704 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa02f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044035070 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa040004c80 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.298+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7fa044044040 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa044067050 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 10 client.17103 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ff737ecb0 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa040004c80 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fa0440046c0 con 0x557ff72f9060 2024-09-16T01:26:03.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 10 client.17103 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 10 client.17103 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 20 client.17103 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff737f320 0x557ff7381710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 4 client.17103 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "87499b9d", "release": true, "await": true}] 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "87499b9d", "release": true, "await": true}) -- 0x557ff72660a0 con 0x557ff737f320 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.299+0000 7fa04ffff640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff737f320 0x557ff7381710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.300+0000 7fa02f7fe640 10 client.17103 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:03.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.300+0000 7fa04ffff640 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff737f320 0x557ff7381710 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1aa9b5d48d4d706 server_cookie=3a86c1ffe03ad355 in_seq=0 out_seq=0 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.346+0000 7fa02f7fe640 1 -- 172.21.15.78:0/1787485176 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x557ff72660a0 con 0x557ff737f320 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.346+0000 7fa02f7fe640 10 client.17103 handle_command_reply: tid=0 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 client.17103 shutdown 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 msgr2=0x557ff730c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa044031bf0 tx=0x7fa044009780 comp rx=0 tx=0).stop 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 shutdown_connections 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557ff737f320 0x557ff7381710 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557ff7304240 0x557ff73091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa02f7fe640 0 client.17103 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ff72f9980 0x557ff730c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 --2- 172.21.15.78:0/1787485176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557ff72f9060 0x557ff730c1e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 >> 172.21.15.78:0/1787485176 conn(0x557ff72f55f0 msgr2=0x557ff72fb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 shutdown_connections 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.347+0000 7fa058b5a740 1 -- 172.21.15.78:0/1787485176 wait complete. 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.348+0000 7fa058b5a740 20 client.17103 trim_cache size 0 max 16384 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:26:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 56, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "87499b9d": { 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 56, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 137.7, 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:26:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:03.352 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.348+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 msgr2=0x7fa0380a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.348+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 0x7fa0380a5870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa048064070 tx=0x7fa048098680 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.348+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 msgr2=0x7fa05019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.348+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eaa0 tx=0x7fa04800ef60 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 shutdown_connections 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fa0380a33b0 0x7fa0380a5870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa050106a80 0x7fa05019cc90 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa050101420 0x7fa050198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 --2- 172.21.15.78:0/3915315397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa050100a80 0x7fa0501983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 >> 172.21.15.78:0/3915315397 conn(0x7fa0500fc7a0 msgr2=0x7fa0500fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 shutdown_connections 2024-09-16T01:26:03.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:03.349+0000 7fa056781640 1 -- 172.21.15.78:0/3915315397 wait complete. 2024-09-16T01:26:03.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:03 smithi148 ceph-mon[35030]: pgmap v479: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-09-16T01:26:03.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:03 smithi193 ceph-mon[35701]: pgmap v479: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-09-16T01:26:04.105 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '87499b9d', total seconds elapsed: 21.3 2024-09-16T01:26:04.105 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-16T01:26:04.105 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:26:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:05.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:05 smithi078 ceph-mon[30937]: pgmap v480: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T01:26:05.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.209+0000 7f46bc556640 1 -- 172.21.15.78:0/1401856999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 msgr2=0x7f46b4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.209+0000 7f46bc556640 1 --2- 172.21.15.78:0/1401856999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4109c90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009be0 tx=0x7f46ac02a6e0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.210+0000 7f46bc556640 1 -- 172.21.15.78:0/1401856999 shutdown_connections 2024-09-16T01:26:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.210+0000 7f46bc556640 1 --2- 172.21.15.78:0/1401856999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 0x7f46b411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.210+0000 7f46bc556640 1 --2- 172.21.15.78:0/1401856999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4109c90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.210+0000 7f46bc556640 1 --2- 172.21.15.78:0/1401856999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.210+0000 7f46bc556640 1 -- 172.21.15.78:0/1401856999 >> 172.21.15.78:0/1401856999 conn(0x7f46b406e140 msgr2=0x7f46b406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:05.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.214+0000 7f46bc556640 1 -- 172.21.15.78:0/1401856999 shutdown_connections 2024-09-16T01:26:05.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.215+0000 7f46bc556640 1 -- 172.21.15.78:0/1401856999 wait complete. 2024-09-16T01:26:05.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.215+0000 7f46bc556640 1 Processor -- start 2024-09-16T01:26:05.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 -- start start 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4114930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 0x7f46b4114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f46b4111330 con 0x7f46b4109450 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f46b41111b0 con 0x7f46b4109830 2024-09-16T01:26:05.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.219+0000 7f46bc556640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f46b41114b0 con 0x7f46b411b140 2024-09-16T01:26:05.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.224+0000 7f46ba2f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:05.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.225+0000 7f46ba2f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52914/0 (socket says 172.21.15.78:52914) 2024-09-16T01:26:05.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.225+0000 7f46ba2f2640 1 -- 172.21.15.78:0/593718446 learned_addr learned my addr 172.21.15.78:0/593718446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:05.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.228+0000 7f46b9af1640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4114930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:05.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.228+0000 7f46baaf3640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 0x7f46b4114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 -- 172.21.15.78:0/593718446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 msgr2=0x7f46b4114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 0x7f46b4114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 -- 172.21.15.78:0/593718446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 msgr2=0x7f46b4114930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4114930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 -- 172.21.15.78:0/593718446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b41155c0 con 0x7f46b4109450 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ba2f2640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9c0 tx=0x7f46b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b000ccc0 con 0x7f46b4109450 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b41bc130 con 0x7f46b4109450 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.229+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b41bc5e0 con 0x7f46b4109450 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.230+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46b0005500 con 0x7f46b4109450 2024-09-16T01:26:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.230+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0010430 con 0x7f46b4109450 2024-09-16T01:26:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.234+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f46b00046c0 con 0x7f46b4109450 2024-09-16T01:26:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.234+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4698005290 con 0x7f46b4109450 2024-09-16T01:26:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.235+0000 7f46ab7fe640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 0x7f46a00a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.236+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9525+0+0 (secure 0 0 0) 0x7f46b00d0370 con 0x7f46b4109450 2024-09-16T01:26:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.241+0000 7f46b9af1640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 0x7f46a00a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:05.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.245+0000 7f46b9af1640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 0x7f46a00a5a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f46b4116180 tx=0x7f46ac000f40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:05.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.245+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f46b009c960 con 0x7f46b4109450 2024-09-16T01:26:05.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:05 smithi148 ceph-mon[35030]: pgmap v480: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T01:26:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:05 smithi193 ceph-mon[35701]: pgmap v480: 129 pgs: 129 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 2.5 MiB/s wr, 259 op/s 2024-09-16T01:26:05.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.658+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4698005850 con 0x7f46b4109450 2024-09-16T01:26:05.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.664+0000 7f46ab7fe640 1 -- 172.21.15.78:0/593718446 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f46b009c260 con 0x7f46b4109450 2024-09-16T01:26:05.665 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:05.666 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:26:05.666 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:26:05.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 msgr2=0x7f46a00a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:05.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 0x7f46a00a5a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f46b4116180 tx=0x7f46ac000f40 comp rx=0 tx=0).stop 2024-09-16T01:26:05.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 msgr2=0x7f46b411a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9c0 tx=0x7f46b000ee80 comp rx=0 tx=0).stop 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 shutdown_connections 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f46a00a3540 0x7f46a00a5a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46b411b140 0x7f46b4114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46b4109830 0x7f46b4114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 --2- 172.21.15.78:0/593718446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46b4109450 0x7f46b411a8b0 secure :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9c0 tx=0x7f46b000ee80 comp rx=0 tx=0).stop 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.667+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 >> 172.21.15.78:0/593718446 conn(0x7f46b406e140 msgr2=0x7f46b4070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.668+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 shutdown_connections 2024-09-16T01:26:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:05.668+0000 7f46bc556640 1 -- 172.21.15.78:0/593718446 wait complete. 2024-09-16T01:26:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 890.0 s, 138.7 tps, lat 1812.165 ms stddev 3692.683 2024-09-16T01:26:06.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:06 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/593718446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:06.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:06 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/593718446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:06.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:06 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/593718446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:07.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:07 smithi148 ceph-mon[35030]: pgmap v481: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-16T01:26:07.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:07 smithi078 ceph-mon[30937]: pgmap v481: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-16T01:26:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:07 smithi193 ceph-mon[35701]: pgmap v481: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 3.2 MiB/s wr, 311 op/s 2024-09-16T01:26:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.866+0000 7f5119f1d640 1 -- 172.21.15.78:0/3668650287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0b2df0 msgr2=0x7f510c0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.866+0000 7f5119f1d640 1 --2- 172.21.15.78:0/3668650287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0b2df0 0x7f510c0b5270 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5114067960 tx=0x7f5114096c80 comp rx=0 tx=0).stop 2024-09-16T01:26:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.869+0000 7f5119f1d640 1 -- 172.21.15.78:0/3668650287 shutdown_connections 2024-09-16T01:26:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.869+0000 7f5119f1d640 1 --2- 172.21.15.78:0/3668650287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0b2df0 0x7f510c0b5270 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.869+0000 7f5119f1d640 1 --2- 172.21.15.78:0/3668650287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f510c0a53b0 0x7f510c0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.869+0000 7f5119f1d640 1 --2- 172.21.15.78:0/3668650287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0a4a90 0x7f510c0a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.869+0000 7f5119f1d640 1 -- 172.21.15.78:0/3668650287 >> 172.21.15.78:0/3668650287 conn(0x7f510c01a110 msgr2=0x7f510c01a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.870+0000 7f5119f1d640 1 -- 172.21.15.78:0/3668650287 shutdown_connections 2024-09-16T01:26:08.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.872+0000 7f5119f1d640 1 -- 172.21.15.78:0/3668650287 wait complete. 2024-09-16T01:26:08.879 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.873+0000 7f5119f1d640 1 Processor -- start 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 -- start start 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f510c0a4a90 0x7f510c140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0b2df0 0x7f510c13af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f510c0aacf0 con 0x7f510c0b2df0 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5113fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5113fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36316/0 (socket says 172.21.15.78:36316) 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5113fff640 1 -- 172.21.15.78:0/181878008 learned_addr learned my addr 172.21.15.78:0/181878008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f510c0aab70 con 0x7f510c0a53b0 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.877+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f510c0aae70 con 0x7f510c0a4a90 2024-09-16T01:26:08.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f511971c640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0b2df0 0x7f510c13af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f5113fff640 1 -- 172.21.15.78:0/181878008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f510c0a4a90 msgr2=0x7f510c140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f5113fff640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f510c0a4a90 0x7f510c140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f5113fff640 1 -- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0b2df0 msgr2=0x7f510c13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f5113fff640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0b2df0 0x7f510c13af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.878+0000 7f5113fff640 1 -- 172.21.15.78:0/181878008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c13b6d0 con 0x7f510c0a53b0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.879+0000 7f5113fff640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f510800e990 tx=0x7f510800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.879+0000 7f5111ffb640 1 -- 172.21.15.78:0/181878008 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510800cbe0 con 0x7f510c0a53b0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.880+0000 7f5111ffb640 1 -- 172.21.15.78:0/181878008 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f510800cd80 con 0x7f510c0a53b0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.880+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510c13b9c0 con 0x7f510c0a53b0 2024-09-16T01:26:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.880+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510c1477d0 con 0x7f510c0a53b0 2024-09-16T01:26:08.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.880+0000 7f5111ffb640 1 -- 172.21.15.78:0/181878008 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5108011470 con 0x7f510c0a53b0 2024-09-16T01:26:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.887+0000 7f5111ffb640 1 -- 172.21.15.78:0/181878008 <== mon.2 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f51080116f0 con 0x7f510c0a53b0 2024-09-16T01:26:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.890+0000 7f511dff5740 1 Processor -- start 2024-09-16T01:26:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.890+0000 7f5111ffb640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 0x7f51040a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.890+0000 7f5111ffb640 1 -- 172.21.15.78:0/181878008 <== mon.2 v2:172.21.15.148:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f51080d1090 con 0x7f510c0a53b0 2024-09-16T01:26:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.890+0000 7f5118f1b640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 0x7f51040a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 -- start start 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a435e0 0x5575b9a4d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5575b99a72f0 con 0x5575b9a435e0 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5575b994f270 con 0x5575b9a4de80 2024-09-16T01:26:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.891+0000 7f511dff5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575b99a32b0 con 0x5575b9a42cc0 2024-09-16T01:26:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.893+0000 7f511971c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.893+0000 7f511971c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:36328/0 (socket says 172.21.15.78:36328) 2024-09-16T01:26:08.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.893+0000 7f511971c640 1 -- 172.21.15.78:0/424525564 learned_addr learned my addr 172.21.15.78:0/424525564 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.894+0000 7f511971c640 1 -- 172.21.15.78:0/424525564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 msgr2=0x5575b9a430a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:26:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.894+0000 7f511971c640 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a430a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.894+0000 7f511971c640 1 -- 172.21.15.78:0/424525564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a435e0 msgr2=0x5575b9a4d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.894+0000 7f511971c640 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a435e0 0x5575b9a4d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.894+0000 7f511971c640 1 -- 172.21.15.78:0/424525564 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575b994e340 con 0x5575b9a4de80 2024-09-16T01:26:08.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.895+0000 7f511971c640 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5114067f00 tx=0x7f5114068ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a653620cbf6a065 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.896+0000 7f50feffd640 1 -- 172.21.15.78:0/424525564 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51140794a0 con 0x5575b9a4de80 2024-09-16T01:26:08.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.896+0000 7f50feffd640 1 -- 172.21.15.78:0/424525564 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511409b040 con 0x5575b9a4de80 2024-09-16T01:26:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.897+0000 7f511dff5740 1 -- 172.21.15.78:0/424525564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 msgr2=0x5575b9a50270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.897+0000 7f511dff5740 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5114067f00 tx=0x7f5114068ef0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.898+0000 7f511dff5740 1 -- 172.21.15.78:0/424525564 shutdown_connections 2024-09-16T01:26:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.898+0000 7f511dff5740 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a4de80 0x5575b9a50270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.898+0000 7f511dff5740 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a435e0 0x5575b9a4d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.898+0000 7f511dff5740 1 --2- 172.21.15.78:0/424525564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.898+0000 7f511dff5740 1 -- 172.21.15.78:0/424525564 >> 172.21.15.78:0/424525564 conn(0x5575b9a3f230 msgr2=0x5575b9a3f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:08.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.899+0000 7f511dff5740 1 -- 172.21.15.78:0/424525564 shutdown_connections 2024-09-16T01:26:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.902+0000 7f511dff5740 1 -- 172.21.15.78:0/424525564 wait complete. 2024-09-16T01:26:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.902+0000 7f511dff5740 1 -- start start 2024-09-16T01:26:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.906+0000 7f511dff5740 1 Processor -- start 2024-09-16T01:26:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a4ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a4b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5575b9a61ab0 con 0x5575b9a4de80 2024-09-16T01:26:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5575b9930290 con 0x5575b9a435e0 2024-09-16T01:26:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5575b9a61c30 con 0x5575b9a42cc0 2024-09-16T01:26:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.910+0000 7f511dff5740 10 client.0 _ll_register_callbacks cb 0x5575b9961140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.911+0000 7f511971c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.911+0000 7f511971c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52942/0 (socket says 172.21.15.78:52942) 2024-09-16T01:26:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.911+0000 7f511971c640 1 -- 172.21.15.78:0/3840176628 learned_addr learned my addr 172.21.15.78:0/3840176628 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f5118f1b640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a4ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f5118f1b640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 0x7f51040a5a50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f510c0b2a00 tx=0x7f5100006c40 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f511971c640 1 -- 172.21.15.78:0/3840176628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 msgr2=0x5575b9a4ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f511971c640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a4ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f511971c640 1 -- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 msgr2=0x5575b9a4b2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f511971c640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a4b2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.912+0000 7f511971c640 1 -- 172.21.15.78:0/3840176628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575b96ba210 con 0x5575b9a4de80 2024-09-16T01:26:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.913+0000 7f511971c640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f51140518b0 tx=0x7f511406a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6252814023fec999 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.913+0000 7f50f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.913+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51140b0060 con 0x5575b9a4de80 2024-09-16T01:26:08.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.913+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f511409b040 con 0x5575b9a4de80 2024-09-16T01:26:08.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.914+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f5114079770 con 0x5575b9a4de80 2024-09-16T01:26:08.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.914+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f51140b7050 con 0x5575b9a4de80 2024-09-16T01:26:08.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.914+0000 7f511dff5740 10 client.17115 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:08.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.914+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575b9786620 con 0x5575b9a4de80 2024-09-16T01:26:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.915+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511409b040 con 0x5575b9a4de80 2024-09-16T01:26:08.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.915+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f51140978a0 con 0x5575b9a4de80 2024-09-16T01:26:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 10 client.17115 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 10 client.17115 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 20 client.17115 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9a6eed0 0x5575b9a71320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 4 client.17115 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.916+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575b99b4a20 con 0x5575b9a6eed0 2024-09-16T01:26:08.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.925+0000 7f5118f1b640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9a6eed0 0x5575b9a71320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.938+0000 7f5118f1b640 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9a6eed0 0x5575b9a71320 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb40e2b274c64013 server_cookie=a1832c06cf74c03d in_seq=0 out_seq=0 2024-09-16T01:26:08.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.939+0000 7f50f3fff640 10 client.17115 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.941+0000 7f50f3fff640 1 -- 172.21.15.78:0/3840176628 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5575b99b4a20 con 0x5575b9a6eed0 2024-09-16T01:26:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.941+0000 7f50f3fff640 10 client.17115 handle_command_reply: tid=0 2024-09-16T01:26:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.941+0000 7f511dff5740 1 client.17115 shutdown 2024-09-16T01:26:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.944+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 msgr2=0x5575b9a4b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.944+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f51140518b0 tx=0x7f511406a6f0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 shutdown_connections 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9a6eed0 0x5575b9a71320 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a4b7f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a4b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 --2- 172.21.15.78:0/3840176628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a4ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 >> 172.21.15.78:0/3840176628 conn(0x5575b9a3f230 msgr2=0x5575b9a44f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:08.948 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.946+0000 7f50f3fff640 0 client.17115 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.947+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 shutdown_connections 2024-09-16T01:26:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.947+0000 7f511dff5740 1 -- 172.21.15.78:0/3840176628 wait complete. 2024-09-16T01:26:08.950 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.948+0000 7f511dff5740 20 client.17115 trim_cache size 0 max 16384 2024-09-16T01:26:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.965+0000 7f511dff5740 1 Processor -- start 2024-09-16T01:26:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.965+0000 7f511dff5740 1 -- start start 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a42cc0 0x5575b9a76c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 0x5575b9a77170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f50f8012af0 con 0x5575b9a4de80 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50f80103c0 con 0x5575b9a42cc0 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511dff5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50f800e230 con 0x5575b9a435e0 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f5113fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 0x5575b9a77170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f5113fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 0x5575b9a77170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:41012/0 (socket says 172.21.15.78:41012) 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f5113fff640 1 -- 172.21.15.78:0/4239385508 learned_addr learned my addr 172.21.15.78:0/4239385508 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f5118f1b640 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a42cc0 0x5575b9a76c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.966+0000 7f511971c640 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.967+0000 7f511971c640 1 -- 172.21.15.78:0/4239385508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 msgr2=0x5575b9a77170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.967+0000 7f511971c640 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 0x5575b9a77170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.967+0000 7f511971c640 1 -- 172.21.15.78:0/4239385508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a42cc0 msgr2=0x5575b9a76c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.967+0000 7f511971c640 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a42cc0 0x5575b9a76c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.967+0000 7f511971c640 1 -- 172.21.15.78:0/4239385508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575b9a719a0 con 0x5575b9a4de80 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.968+0000 7f511971c640 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47fd0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5114067960 tx=0x7f5114096c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359ded25500177a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.968+0000 7f50fe7fc640 1 -- 172.21.15.78:0/4239385508 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51140b0070 con 0x5575b9a4de80 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.968+0000 7f50fe7fc640 1 -- 172.21.15.78:0/4239385508 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511409b040 con 0x5575b9a4de80 2024-09-16T01:26:08.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.968+0000 7f50fe7fc640 1 -- 172.21.15.78:0/4239385508 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51140ab9d0 con 0x5575b9a4de80 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 -- 172.21.15.78:0/4239385508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 msgr2=0x5575b9a47fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47fd0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5114067960 tx=0x7f5114096c80 comp rx=0 tx=0).stop 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 -- 172.21.15.78:0/4239385508 shutdown_connections 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47fd0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a435e0 0x5575b9a77170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 --2- 172.21.15.78:0/4239385508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a42cc0 0x5575b9a76c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 -- 172.21.15.78:0/4239385508 >> 172.21.15.78:0/4239385508 conn(0x5575b9a3f230 msgr2=0x5575b9a4cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 -- 172.21.15.78:0/4239385508 shutdown_connections 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.969+0000 7f511dff5740 1 -- 172.21.15.78:0/4239385508 wait complete. 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.970+0000 7f511dff5740 1 -- start start 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.970+0000 7f511dff5740 1 Processor -- start 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f511dff5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a67500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f511dff5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a47860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f511dff5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f5118f1b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a67500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f5118f1b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a67500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:41024/0 (socket says 172.21.15.78:41024) 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f5118f1b640 1 -- 172.21.15.78:0/890086125 learned_addr learned my addr 172.21.15.78:0/890086125 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f5113fff640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a47860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f511971c640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.971+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f50f800e880 con 0x5575b9a4de80 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50f800fcf0 con 0x5575b9a435e0 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50f800ef50 con 0x5575b9a42cc0 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f511dff5740 10 client.0 _ll_register_callbacks cb 0x7f5114068e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 -- 172.21.15.78:0/890086125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 msgr2=0x5575b9a67500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a67500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 -- 172.21.15.78:0/890086125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 msgr2=0x5575b9a47da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575b9a48390 con 0x5575b9a435e0 2024-09-16T01:26:08.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f511971c640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f5113fff640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a47860 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f51080ce3a0 tx=0x7f510809cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1caf8e38a086a68e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.972+0000 7f50f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080b8020 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mon.2 v2:172.21.15.148:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f51080a58b0 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f511409b040 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mon.2 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f51080aa050 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 10 client.26354 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575b9ad99e0 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mon.2 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511409b040 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mon.2 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f510800f040 con 0x5575b9a435e0 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 10 client.26354 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 10 client.26354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 20 client.26354 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9ada1e0 0x5575b9adc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 4 client.26354 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575b99aed70 con 0x5575b9ada1e0 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.973+0000 7f5118f1b640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9ada1e0 0x5575b9adc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.974+0000 7f50f3fff640 10 client.26354 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:08.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:08.974+0000 7f5118f1b640 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9ada1e0 0x5575b9adc600 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fedb470de0c28b9a server_cookie=6f53078f2ae016c5 in_seq=0 out_seq=0 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.161+0000 7f50f3fff640 1 -- 172.21.15.78:0/890086125 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575b99aed70 con 0x5575b9ada1e0 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.161+0000 7f50f3fff640 10 client.26354 handle_command_reply: tid=0 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.161+0000 7f511dff5740 1 client.26354 shutdown 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 msgr2=0x5575b9a47860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a47860 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f51080ce3a0 tx=0x7f510809cb00 comp rx=0 tx=0).stop 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 shutdown_connections 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5575b9ada1e0 0x5575b9adc600 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f50f3fff640 0 client.26354 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575b9a4de80 0x5575b9a47da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5575b9a435e0 0x5575b9a47860 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 --2- 172.21.15.78:0/890086125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575b9a42cc0 0x5575b9a67500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 >> 172.21.15.78:0/890086125 conn(0x5575b9a3f230 msgr2=0x5575b9a4ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 shutdown_connections 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 1 -- 172.21.15.78:0/890086125 wait complete. 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.162+0000 7f511dff5740 20 client.26354 trim_cache size 0 max 16384 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:26:09.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:26:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 msgr2=0x7f51040a5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 0x7f51040a5a50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f510c0b2a00 tx=0x7f5100006c40 comp rx=0 tx=0).stop 2024-09-16T01:26:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 msgr2=0x7f510c140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f510800e990 tx=0x7f510800ee50 comp rx=0 tx=0).stop 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 shutdown_connections 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f51040a3590 0x7f51040a5a50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f510c0b2df0 0x7f510c13af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f510c0a53b0 0x7f510c140e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 --2- 172.21.15.78:0/181878008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f510c0a4a90 0x7f510c140910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 >> 172.21.15.78:0/181878008 conn(0x7f510c01a110 msgr2=0x7f510c0b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 shutdown_connections 2024-09-16T01:26:09.168 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:09.163+0000 7f5119f1d640 1 -- 172.21.15.78:0/181878008 wait complete. 2024-09-16T01:26:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:09 smithi148 ceph-mon[35030]: pgmap v482: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 3.4 MiB/s wr, 289 op/s 2024-09-16T01:26:09.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:09 smithi148 ceph-mon[35030]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T01:26:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:09 smithi078 ceph-mon[30937]: pgmap v482: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 3.4 MiB/s wr, 289 op/s 2024-09-16T01:26:09.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:09 smithi078 ceph-mon[30937]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T01:26:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:09 smithi193 ceph-mon[35701]: pgmap v482: 129 pgs: 129 active+clean; 3.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 3.4 MiB/s wr, 289 op/s 2024-09-16T01:26:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:09 smithi193 ceph-mon[35701]: osdmap e114: 12 total, 12 up, 12 in 2024-09-16T01:26:10.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 895.0 s, 367.6 tps, lat 279.839 ms stddev 404.245 2024-09-16T01:26:11.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:11 smithi148 ceph-mon[35030]: pgmap v484: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 460 op/s 2024-09-16T01:26:11.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:11 smithi078 ceph-mon[30937]: pgmap v484: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 460 op/s 2024-09-16T01:26:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:11 smithi193 ceph-mon[35701]: pgmap v484: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 460 op/s 2024-09-16T01:26:12.561 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-16T01:26:12.562 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-16T01:26:13.029 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:13.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:13 smithi078 ceph-mon[30937]: pgmap v485: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 537 op/s 2024-09-16T01:26:13.196 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:13 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:26:13.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:13 smithi148 ceph-mon[35030]: pgmap v485: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 537 op/s 2024-09-16T01:26:13.387 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:13 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:26:13.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:13 smithi193 ceph-mon[35701]: pgmap v485: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 537 op/s 2024-09-16T01:26:13.550 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:13 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.204+0000 7f4a05dcc640 1 -- 172.21.15.78:0/1318712301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a00109450 msgr2=0x7f4a00110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.204+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/1318712301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a00109450 0x7f4a00110250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49f001a910 tx=0x7f49f003b470 comp rx=0 tx=0).stop 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.206+0000 7f4a05dcc640 1 -- 172.21.15.78:0/1318712301 shutdown_connections 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.206+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/1318712301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a0011b140 0x7f4a0011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.206+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/1318712301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109830 0x7f4a00109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.206+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/1318712301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a00109450 0x7f4a00110250 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.206+0000 7f4a05dcc640 1 -- 172.21.15.78:0/1318712301 >> 172.21.15.78:0/1318712301 conn(0x7f4a0006e140 msgr2=0x7f4a0006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.212+0000 7f4a05dcc640 1 -- 172.21.15.78:0/1318712301 shutdown_connections 2024-09-16T01:26:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.213+0000 7f4a05dcc640 1 -- 172.21.15.78:0/1318712301 wait complete. 2024-09-16T01:26:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.214+0000 7f4a05dcc640 1 Processor -- start 2024-09-16T01:26:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 -- start start 2024-09-16T01:26:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a00109830 0x7f4a00116a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a0011b140 0x7f4a00116fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a001133e0 con 0x7f4a00109450 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00113260 con 0x7f4a0011b140 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.217+0000 7f4a05dcc640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00113560 con 0x7f4a00109830 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a0011b140 0x7f4a00116fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43340/0 (socket says 172.21.15.78:43340) 2024-09-16T01:26:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 -- 172.21.15.78:0/3320067069 learned_addr learned my addr 172.21.15.78:0/3320067069 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 -- 172.21.15.78:0/3320067069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a00109830 msgr2=0x7f4a00116a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:26:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a00109830 0x7f4a00116a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 -- 172.21.15.78:0/3320067069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a0011b140 msgr2=0x7f4a00116fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a0011b140 0x7f4a00116fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.220+0000 7f49ff7fe640 1 -- 172.21.15.78:0/3320067069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a00117870 con 0x7f4a00109450 2024-09-16T01:26:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.221+0000 7f49ff7fe640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f49f001aa30 tx=0x7f49f0004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.222+0000 7f49fd7fa640 1 -- 172.21.15.78:0/3320067069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f004f070 con 0x7f4a00109450 2024-09-16T01:26:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.222+0000 7f49fd7fa640 1 -- 172.21.15.78:0/3320067069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f0040c00 con 0x7f4a00109450 2024-09-16T01:26:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.222+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a001ab1d0 con 0x7f4a00109450 2024-09-16T01:26:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.222+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a001ab6e0 con 0x7f4a00109450 2024-09-16T01:26:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.226+0000 7f4a081a5740 1 Processor -- start 2024-09-16T01:26:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.226+0000 7f49fd7fa640 1 -- 172.21.15.78:0/3320067069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0046030 con 0x7f4a00109450 2024-09-16T01:26:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.226+0000 7f4a081a5740 1 -- start start 2024-09-16T01:26:14.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add49400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add49940 0x5587add4bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5587adca42a0 con 0x5587add49940 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5587adc4bb10 con 0x5587add3f080 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.227+0000 7f4a081a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587adca0260 con 0x5587add3e760 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.229+0000 7f49fffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.233+0000 7f49fffff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:40216/0 (socket says 172.21.15.78:40216) 2024-09-16T01:26:14.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.233+0000 7f49fffff640 1 -- 172.21.15.78:0/1490351301 learned_addr learned my addr 172.21.15.78:0/1490351301 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.249+0000 7f49fffff640 1 -- 172.21.15.78:0/1490351301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 msgr2=0x5587add49400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:26:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.249+0000 7f49fffff640 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add49400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.249+0000 7f49fffff640 1 -- 172.21.15.78:0/1490351301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add49940 msgr2=0x5587add4bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.247+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add49940 0x5587add4bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.251+0000 7f49fffff640 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add49940 0x5587add4bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.251+0000 7f49fffff640 1 -- 172.21.15.78:0/1490351301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587ada15b90 con 0x5587add3e760 2024-09-16T01:26:14.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.252+0000 7f49fd7fa640 1 -- 172.21.15.78:0/3320067069 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f49f0052070 con 0x7f4a00109450 2024-09-16T01:26:14.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.253+0000 7f49fd7fa640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 0x7f49ec0a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.253+0000 7f49fd7fa640 1 -- 172.21.15.78:0/3320067069 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f49f0101df0 con 0x7f4a00109450 2024-09-16T01:26:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.255+0000 7f49fffff640 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007ce0 tx=0x7f49f800c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1675a0c82f476c26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.256+0000 7f49f5ffb640 1 -- 172.21.15.78:0/1490351301 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8015020 con 0x5587add3e760 2024-09-16T01:26:14.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.256+0000 7f49f5ffb640 1 -- 172.21.15.78:0/1490351301 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f800ce50 con 0x5587add3e760 2024-09-16T01:26:14.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.256+0000 7f49ff7fe640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 0x7f49ec0a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.257+0000 7f4a081a5740 1 -- 172.21.15.78:0/1490351301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 msgr2=0x5587add3eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.257+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007ce0 tx=0x7f49f800c540 comp rx=0 tx=0).stop 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.260+0000 7f4a081a5740 1 -- 172.21.15.78:0/1490351301 shutdown_connections 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.260+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add49940 0x5587add4bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.260+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add49400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.260+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1490351301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3e760 0x5587add3eb40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.260+0000 7f4a081a5740 1 -- 172.21.15.78:0/1490351301 >> 172.21.15.78:0/1490351301 conn(0x5587add3acf0 msgr2=0x5587add3b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.261+0000 7f49ff7fe640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 0x7f49ec0a5a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5587add3bb50 tx=0x7f49f004b000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.261+0000 7f4a081a5740 1 -- 172.21.15.78:0/1490351301 shutdown_connections 2024-09-16T01:26:14.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.265+0000 7f4a081a5740 1 -- 172.21.15.78:0/1490351301 wait complete. 2024-09-16T01:26:14.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.266+0000 7f4a081a5740 1 -- start start 2024-09-16T01:26:14.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.266+0000 7f4a081a5740 1 Processor -- start 2024-09-16T01:26:14.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.266+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.266+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add46cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add47230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f4a081a5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5587add4c560 con 0x5587add3e760 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f4a081a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5587adc2c6e0 con 0x5587add3f080 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f4a081a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587add4c6e0 con 0x5587add49940 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f4a081a5740 10 client.0 _ll_register_callbacks cb 0x5587adc50a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f49fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f49fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add467b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43362/0 (socket says 172.21.15.78:43362) 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f49fffff640 1 -- 172.21.15.78:0/1522729080 learned_addr learned my addr 172.21.15.78:0/1522729080 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f49fffff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add47230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.267+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add46cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49fffff640 1 -- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 msgr2=0x5587add46cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49fffff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add46cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49fffff640 1 -- 172.21.15.78:0/1522729080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 msgr2=0x5587add467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49fffff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49fffff640 1 -- 172.21.15.78:0/1522729080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587add55ab0 con 0x5587add49940 2024-09-16T01:26:14.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.268+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add46cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.270+0000 7f49fffff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add47230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007cb0 tx=0x7f49f80075d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5468d0a509121be0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.270+0000 7f49deffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.270+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8015030 con 0x5587add49940 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.271+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f800ce50 con 0x5587add49940 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.271+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f49f8011610 con 0x5587add49940 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.271+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f49f8011810 con 0x5587add49940 2024-09-16T01:26:14.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.271+0000 7f4a081a5740 10 client.26404 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:14.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.271+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587add58440 con 0x5587add49940 2024-09-16T01:26:14.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.272+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f800ce50 con 0x5587add49940 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f49f8010050 con 0x5587add49940 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 10 client.26404 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 10 client.26404 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 20 client.26404 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587add59d50 0x5587add5c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 4 client.26404 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587adcb19d0 con 0x5587add59d50 2024-09-16T01:26:14.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.273+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587add59d50 0x5587add5c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.276+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587add59d50 0x5587add5c190 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3230d753811881 server_cookie=7397b1090972866a in_seq=0 out_seq=0 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.276+0000 7f49deffd640 10 client.26404 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f49deffd640 1 -- 172.21.15.78:0/1522729080 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5587adcb19d0 con 0x5587add59d50 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f49deffd640 10 client.26404 handle_command_reply: tid=0 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 client.26404 shutdown 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 msgr2=0x5587add47230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add47230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49f8007cb0 tx=0x7f49f80075d0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 shutdown_connections 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587add59d50 0x5587add5c190 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add47230 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3f080 0x5587add46cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1522729080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 >> 172.21.15.78:0/1522729080 conn(0x5587add3acf0 msgr2=0x5587add409d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.277+0000 7f49deffd640 0 client.26404 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.280+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 shutdown_connections 2024-09-16T01:26:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.280+0000 7f4a081a5740 1 -- 172.21.15.78:0/1522729080 wait complete. 2024-09-16T01:26:14.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.280+0000 7f4a081a5740 20 client.26404 trim_cache size 0 max 16384 2024-09-16T01:26:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.311+0000 7f4a081a5740 1 Processor -- start 2024-09-16T01:26:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 -- start start 2024-09-16T01:26:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3f080 0x5587add408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add61aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f49e0012b00 con 0x5587add3f080 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f49e00103d0 con 0x5587add3e760 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f4a081a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49e000e240 con 0x5587add49940 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f49fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f49fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:46458/0 (socket says 172.21.15.78:46458) 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f49fffff640 1 -- 172.21.15.78:0/1187758840 learned_addr learned my addr 172.21.15.78:0/1187758840 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:14.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f49fffff640 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add61aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.312+0000 7f49f7fff640 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3f080 0x5587add408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 -- 172.21.15.78:0/1187758840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 msgr2=0x5587add61aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add61aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 -- 172.21.15.78:0/1187758840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3f080 msgr2=0x5587add408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3f080 0x5587add408c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 -- 172.21.15.78:0/1187758840 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587add36260 con 0x5587add3e760 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49fffff640 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49f800cb40 tx=0x7f49f800c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7990f04e4f1b68e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49f57fa640 1 -- 172.21.15.78:0/1187758840 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8015070 con 0x5587add3e760 2024-09-16T01:26:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.313+0000 7f49f57fa640 1 -- 172.21.15.78:0/1187758840 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f80114b0 con 0x5587add3e760 2024-09-16T01:26:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.315+0000 7f4a081a5740 1 -- 172.21.15.78:0/1187758840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 msgr2=0x5587add47080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.315+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49f800cb40 tx=0x7f49f800c540 comp rx=0 tx=0).stop 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 -- 172.21.15.78:0/1187758840 shutdown_connections 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add49940 0x5587add61aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3f080 0x5587add408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 --2- 172.21.15.78:0/1187758840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add3e760 0x5587add47080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 -- 172.21.15.78:0/1187758840 >> 172.21.15.78:0/1187758840 conn(0x5587add3acf0 msgr2=0x5587add401b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.316+0000 7f4a081a5740 1 -- 172.21.15.78:0/1187758840 shutdown_connections 2024-09-16T01:26:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.318+0000 7f4a081a5740 1 -- 172.21.15.78:0/1187758840 wait complete. 2024-09-16T01:26:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 -- start start 2024-09-16T01:26:14.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 Processor -- start 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add52160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 0x5587add43590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add49940 0x5587add43ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f49e000e890 con 0x5587add3e760 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f49e000fd00 con 0x5587add49940 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.319+0000 7f4a081a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49e000ef60 con 0x5587add3f080 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.332+0000 7f4a081a5740 10 client.0 _ll_register_callbacks cb 0x7f49f8010c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.333+0000 7f49fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add52160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.333+0000 7f49fffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add52160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43380/0 (socket says 172.21.15.78:43380) 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.333+0000 7f49fffff640 1 -- 172.21.15.78:0/4236359528 learned_addr learned my addr 172.21.15.78:0/4236359528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.333+0000 7f49fffff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add49940 0x5587add43ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.334+0000 7f49f7fff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 0x5587add43590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.338+0000 7f49f7fff640 1 -- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add49940 msgr2=0x5587add43ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.338+0000 7f49f7fff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add49940 0x5587add43ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.338+0000 7f49f7fff640 1 -- 172.21.15.78:0/4236359528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 msgr2=0x5587add52160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:14.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.338+0000 7f49f7fff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add52160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.338+0000 7f49f7fff640 1 -- 172.21.15.78:0/4236359528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587add3bda0 con 0x5587add3f080 2024-09-16T01:26:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.340+0000 7f49f7fff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 0x5587add43590 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x5587add47df0 tx=0x7f49e8016790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a3707d57cec49d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.340+0000 7f49deffd640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:26:14.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.340+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e801d430 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.340+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f80114b0 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.340+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9613+0+0 (secure 0 0 0) 0x7f49e801d5d0 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.341+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f49e802b050 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.341+0000 7f4a081a5740 10 client.26410 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.341+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587adde6950 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.342+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f80114b0 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f49e801c070 con 0x5587add3f080 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 10 client.26410 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 10 client.26410 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 20 client.26410 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587adde70e0 0x5587adde9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 4 client.26410 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-16T01:26:14.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.343+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x5587adcabd20 con 0x5587adde70e0 2024-09-16T01:26:14.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.358+0000 7f49fffff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587adde70e0 0x5587adde9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:14.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.361+0000 7f49fffff640 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587adde70e0 0x5587adde9500 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd86edb574c2641 server_cookie=51601c2d34348613 in_seq=0 out_seq=0 2024-09-16T01:26:14.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.361+0000 7f49deffd640 10 client.26410 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:14.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f49deffd640 1 -- 172.21.15.78:0/4236359528 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5587adcabd20 con 0x5587adde70e0 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f49deffd640 10 client.26410 handle_command_reply: tid=0 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 client.26410 shutdown 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 msgr2=0x5587add43590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 0x5587add43590 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x5587add47df0 tx=0x7f49e8016790 comp rx=0 tx=0).stop 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 shutdown_connections 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x5587adde70e0 0x5587adde9500 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5587add49940 0x5587add43ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f49deffd640 0 client.26410 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587add3f080 0x5587add43590 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 --2- 172.21.15.78:0/4236359528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587add3e760 0x5587add52160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 >> 172.21.15.78:0/4236359528 conn(0x5587add3acf0 msgr2=0x5587add409d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 shutdown_connections 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 1 -- 172.21.15.78:0/4236359528 wait complete. 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.647+0000 7f4a081a5740 20 client.26410 trim_cache size 0 max 16384 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 58, 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "1f1d21e5": { 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 58, 2024-09-16T01:26:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 137.7, 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "QUIESCED", 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:14.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 msgr2=0x7f49ec0a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 0x7f49ec0a5a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x5587add3bb50 tx=0x7f49f004b000 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 msgr2=0x7f4a0011a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f49f001aa30 tx=0x7f49f0004cf0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 shutdown_connections 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f49ec0a35c0 0x7f49ec0a5a80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a0011b140 0x7f4a00116fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a00109830 0x7f4a00116a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 --2- 172.21.15.78:0/3320067069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a00109450 0x7f4a0011a9d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 >> 172.21.15.78:0/3320067069 conn(0x7f4a0006e140 msgr2=0x7f4a00070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 shutdown_connections 2024-09-16T01:26:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:14.648+0000 7f4a05dcc640 1 -- 172.21.15.78:0/3320067069 wait complete. 2024-09-16T01:26:15.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:15 smithi148 ceph-mon[35030]: pgmap v486: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 539 op/s 2024-09-16T01:26:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:26:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:26:15.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:26:15.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:15 smithi078 ceph-mon[30937]: pgmap v486: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 539 op/s 2024-09-16T01:26:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:26:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:26:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:26:15.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:15 smithi193 ceph-mon[35701]: pgmap v486: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 539 op/s 2024-09-16T01:26:15.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:26:15.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:26:15.549 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:26:15.935 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 900.0 s, 282.0 tps, lat 236.922 ms stddev 257.079 2024-09-16T01:26:16.166 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1f1d21e5', quiesce took 12.1 seconds. Will release after: 5.6 2024-09-16T01:26:17.385 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:17 smithi148 ceph-mon[35030]: pgmap v487: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 6.7 MiB/s wr, 552 op/s 2024-09-16T01:26:17.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:17 smithi078 ceph-mon[30937]: pgmap v487: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 6.7 MiB/s wr, 552 op/s 2024-09-16T01:26:17.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:17 smithi193 ceph-mon[35701]: pgmap v487: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 6.7 MiB/s wr, 552 op/s 2024-09-16T01:26:19.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:19 smithi148 ceph-mon[35030]: pgmap v488: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-09-16T01:26:19.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:19 smithi078 ceph-mon[30937]: pgmap v488: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-09-16T01:26:19.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:19 smithi193 ceph-mon[35701]: pgmap v488: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-09-16T01:26:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:progress: 905.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-16T01:26:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:21 smithi148 ceph-mon[35030]: pgmap v489: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-09-16T01:26:21.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:21 smithi148 ceph-mon[35030]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T01:26:21.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:21 smithi078 ceph-mon[30937]: pgmap v489: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-09-16T01:26:21.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:21 smithi078 ceph-mon[30937]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T01:26:21.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:21 smithi193 ceph-mon[35701]: pgmap v489: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 5.2 MiB/s wr, 398 op/s 2024-09-16T01:26:21.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:21 smithi193 ceph-mon[35701]: osdmap e115: 12 total, 12 up, 12 in 2024-09-16T01:26:21.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:26:22.030 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- 172.21.15.78:0/3994229548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5844109090 msgr2=0x7f5844110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 --2- 172.21.15.78:0/3994229548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5844109090 0x7f5844110be0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f583c009ca0 tx=0x7f583c02a8c0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- 172.21.15.78:0/3994229548 shutdown_connections 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 --2- 172.21.15.78:0/3994229548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5844109090 0x7f5844110be0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 --2- 172.21.15.78:0/3994229548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58441048f0 0x7f5844108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 --2- 172.21.15.78:0/3994229548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f5844104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- 172.21.15.78:0/3994229548 >> 172.21.15.78:0/3994229548 conn(0x7f58440ff970 msgr2=0x7f5844101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- 172.21.15.78:0/3994229548 shutdown_connections 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- 172.21.15.78:0/3994229548 wait complete. 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 Processor -- start 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.256+0000 7f584ae5e640 1 -- start start 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f584419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5844109090 0x7f584419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f58441140b0 con 0x7f584419edb0 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5844113f30 con 0x7f5844103f40 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5844114230 con 0x7f5844109090 2024-09-16T01:26:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43406/0 (socket says 172.21.15.78:43406) 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 -- 172.21.15.78:0/1057892028 learned_addr learned my addr 172.21.15.78:0/1057892028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5843fff640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5844109090 0x7f584419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5848bfa640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f584419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 -- 172.21.15.78:0/1057892028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5844109090 msgr2=0x7f584419aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5844109090 0x7f584419aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 -- 172.21.15.78:0/1057892028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 msgr2=0x7f584419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f584419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 -- 172.21.15.78:0/1057892028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584419f8a0 con 0x7f584419edb0 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5848bfa640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f584419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f58493fb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f583c031a20 tx=0x7f583c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5841ffb640 1 -- 172.21.15.78:0/1057892028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c003c30 con 0x7f584419edb0 2024-09-16T01:26:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5841ffb640 1 -- 172.21.15.78:0/1057892028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583c003dd0 con 0x7f584419edb0 2024-09-16T01:26:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584419fb30 con 0x7f584419edb0 2024-09-16T01:26:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.257+0000 7f5841ffb640 1 -- 172.21.15.78:0/1057892028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c02fc40 con 0x7f584419edb0 2024-09-16T01:26:22.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.258+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58441a7380 con 0x7f584419edb0 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f5841ffb640 1 -- 172.21.15.78:0/1057892028 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f583c004050 con 0x7f584419edb0 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 Processor -- start 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 -- start start 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df4af0 0x557f24df4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 0x557f24e020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557f24d592f0 con 0x557f24df5410 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557f24d01270 con 0x557f24dffcb0 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f584dbf8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557f24d552b0 con 0x557f24df4af0 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f5848bfa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f5848bfa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43408/0 (socket says 172.21.15.78:43408) 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f5848bfa640 1 -- 172.21.15.78:0/3085292790 learned_addr learned my addr 172.21.15.78:0/3085292790 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.260+0000 7f5843fff640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df4af0 0x557f24df4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5843fff640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 0x557f24e020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5841ffb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 0x7f582c0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 -- 172.21.15.78:0/3085292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df4af0 msgr2=0x557f24df4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df4af0 0x557f24df4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 -- 172.21.15.78:0/3085292790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 msgr2=0x557f24e020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 0x557f24e020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 -- 172.21.15.78:0/3085292790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f24d00340 con 0x557f24df5410 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5843fff640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 0x557f24e020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f58493fb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 0x7f582c0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5848bfa640 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5834009c30 tx=0x7f5834010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12b783bf51a3844 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f5841ffb640 1 -- 172.21.15.78:0/1057892028 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f583c0f1290 con 0x7f584419edb0 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f582affd640 1 -- 172.21.15.78:0/3085292790 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834004030 con 0x557f24df5410 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f582affd640 1 -- 172.21.15.78:0/3085292790 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58340041d0 con 0x557f24df5410 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.261+0000 7f582affd640 1 -- 172.21.15.78:0/3085292790 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834012650 con 0x557f24df5410 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- 172.21.15.78:0/3085292790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 msgr2=0x557f24dff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5834009c30 tx=0x7f5834010740 comp rx=0 tx=0).stop 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f58493fb640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 0x7f582c0a59a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b9780 tx=0x7f583c090150 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- 172.21.15.78:0/3085292790 shutdown_connections 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24dffcb0 0x557f24e020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df5410 0x557f24dff770 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 --2- 172.21.15.78:0/3085292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df4af0 0x557f24df4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- 172.21.15.78:0/3085292790 >> 172.21.15.78:0/3085292790 conn(0x557f24df1060 msgr2=0x557f24df1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- 172.21.15.78:0/3085292790 shutdown_connections 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- 172.21.15.78:0/3085292790 wait complete. 2024-09-16T01:26:22.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 -- start start 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.262+0000 7f584dbf8740 1 Processor -- start 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24dfcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x557f24e02880 con 0x557f24dffcb0 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557f24ce2290 con 0x557f24df4af0 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557f24e02a00 con 0x557f24df5410 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 10 client.0 _ll_register_callbacks cb 0x557f24d13140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43410/0 (socket says 172.21.15.78:43410) 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 -- 172.21.15.78:0/2978875563 learned_addr learned my addr 172.21.15.78:0/2978875563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5848bfa640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24dfcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 -- 172.21.15.78:0/2978875563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 msgr2=0x557f24dfd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 -- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 msgr2=0x557f24dfcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24dfcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 -- 172.21.15.78:0/2978875563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f24a6c210 con 0x557f24dffcb0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5843fff640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f583800cee0 tx=0x7f583800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94dc714955b3bffc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5823fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380142d0 con 0x557f24dffcb0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f58380040f0 con 0x557f24dffcb0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58340041d0 con 0x557f24dffcb0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.263+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5838025050 con 0x557f24dffcb0 2024-09-16T01:26:22.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 10 client.17142 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557f24b38620 con 0x557f24dffcb0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58340041d0 con 0x557f24dffcb0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5838012450 con 0x557f24dffcb0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 10 client.17142 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 10 client.17142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 20 client.17142 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e0fcd0 0x557f24e120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 4 client.17142 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f24d66a20 con 0x557f24e0fcd0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.264+0000 7f5848bfa640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e0fcd0 0x557f24e120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.265+0000 7f5823fff640 10 client.17142 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.265+0000 7f5848bfa640 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e0fcd0 0x557f24e120c0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edacb8ff969bfe49 server_cookie=d07d523f6553ab34 in_seq=0 out_seq=0 2024-09-16T01:26:22.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.265+0000 7f5823fff640 1 -- 172.21.15.78:0/2978875563 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f24d66a20 con 0x557f24e0fcd0 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.265+0000 7f5823fff640 10 client.17142 handle_command_reply: tid=0 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.265+0000 7f584dbf8740 1 client.17142 shutdown 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 msgr2=0x557f24dfd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f583800cee0 tx=0x7f583800cad0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 shutdown_connections 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e0fcd0 0x557f24e120c0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24dfd5d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f5823fff640 0 client.17142 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 --2- 172.21.15.78:0/2978875563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24dfcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 >> 172.21.15.78:0/2978875563 conn(0x557f24df1060 msgr2=0x557f24df6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 shutdown_connections 2024-09-16T01:26:22.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 1 -- 172.21.15.78:0/2978875563 wait complete. 2024-09-16T01:26:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.266+0000 7f584dbf8740 20 client.17142 trim_cache size 0 max 16384 2024-09-16T01:26:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 Processor -- start 2024-09-16T01:26:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 -- start start 2024-09-16T01:26:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24df7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f58441b3a20 con 0x557f24dffcb0 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f58441b12f0 con 0x557f24df4af0 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f584dbf8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f58441af160 con 0x557f24df5410 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43426/0 (socket says 172.21.15.78:43426) 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.280+0000 7f5843fff640 1 -- 172.21.15.78:0/484141960 learned_addr learned my addr 172.21.15.78:0/484141960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24df7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5848bfa640 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 -- 172.21.15.78:0/484141960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 msgr2=0x557f24dfd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 -- 172.21.15.78:0/484141960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 msgr2=0x557f24df7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24df7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 -- 172.21.15.78:0/484141960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f24e0e870 con 0x557f24dffcb0 2024-09-16T01:26:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f5843fff640 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5838005fd0 tx=0x7f583800c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db4b7b9d04312f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f582a7fc640 1 -- 172.21.15.78:0/484141960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380140d0 con 0x557f24dffcb0 2024-09-16T01:26:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f582a7fc640 1 -- 172.21.15.78:0/484141960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5838014270 con 0x557f24dffcb0 2024-09-16T01:26:22.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.281+0000 7f582a7fc640 1 -- 172.21.15.78:0/484141960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838003850 con 0x557f24dffcb0 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- 172.21.15.78:0/484141960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 msgr2=0x557f24e179b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5838005fd0 tx=0x7f583800c3b0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- 172.21.15.78:0/484141960 shutdown_connections 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24dffcb0 0x557f24e179b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24df5410 0x557f24dfd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 --2- 172.21.15.78:0/484141960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df4af0 0x557f24df7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- 172.21.15.78:0/484141960 >> 172.21.15.78:0/484141960 conn(0x557f24df1060 msgr2=0x557f24dfebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- 172.21.15.78:0/484141960 shutdown_connections 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- 172.21.15.78:0/484141960 wait complete. 2024-09-16T01:26:22.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 -- start start 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.282+0000 7f584dbf8740 1 Processor -- start 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 0x557f24df96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24dffcb0 0x557f24df9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f58441af7b0 con 0x557f24df4af0 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f58441b0c20 con 0x557f24df5410 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f58441afe80 con 0x557f24dffcb0 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 10 client.0 _ll_register_callbacks cb 0x7f583800c790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43432/0 (socket says 172.21.15.78:43432) 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 -- 172.21.15.78:0/349447663 learned_addr learned my addr 172.21.15.78:0/349447663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24dffcb0 0x557f24df9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5848bfa640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 0x557f24df96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 -- 172.21.15.78:0/349447663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24dffcb0 msgr2=0x557f24df9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24dffcb0 0x557f24df9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 -- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 msgr2=0x557f24df96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 0x557f24df96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 -- 172.21.15.78:0/349447663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f24dfa250 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5848bfa640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 0x557f24df96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5843fff640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f583802e040 tx=0x7f5838005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b8c81869cfc129 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5823fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838040060 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f5838012a90 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5838014270 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.283+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5838030040 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 10 client.17148 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557f24e7a7e0 con 0x557f24df4af0 2024-09-16T01:26:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5838012c90 con 0x557f24df4af0 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5838039040 con 0x557f24df4af0 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 10 client.17148 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 10 client.17148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 20 client.17148 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e7aed0 0x557f24e7d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 4 client.17148 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5838014270 con 0x557f24e7aed0 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.284+0000 7f5848bfa640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e7aed0 0x557f24e7d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.285+0000 7f5823fff640 10 client.17148 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.285+0000 7f5848bfa640 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e7aed0 0x557f24e7d2f0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ea44cdb6ed4f838 server_cookie=4a016c781eec1532 in_seq=0 out_seq=0 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f5823fff640 1 -- 172.21.15.78:0/349447663 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5838014270 con 0x557f24e7aed0 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f5823fff640 10 client.17148 handle_command_reply: tid=0 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 client.17148 shutdown 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 msgr2=0x557f24e08280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f583802e040 tx=0x7f5838005f30 comp rx=0 tx=0).stop 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 shutdown_connections 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x557f24e7aed0 0x557f24e7d2f0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f24dffcb0 0x557f24df9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f24df5410 0x557f24df96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f5823fff640 0 client.17148 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 --2- 172.21.15.78:0/349447663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557f24df4af0 0x557f24e08280 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.293+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 >> 172.21.15.78:0/349447663 conn(0x557f24df1060 msgr2=0x557f24dfebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 shutdown_connections 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584dbf8740 1 -- 172.21.15.78:0/349447663 wait complete. 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584dbf8740 20 client.17148 trim_cache size 0 max 16384 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:26:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:26:22.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:26:22.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 msgr2=0x7f582c0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 0x7f582c0a59a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b9780 tx=0x7f583c090150 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 msgr2=0x7f584419f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f583c031a20 tx=0x7f583c031a50 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 shutdown_connections 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f582c0a34e0 0x7f582c0a59a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f584419edb0 0x7f584419f260 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5844109090 0x7f584419aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 --2- 172.21.15.78:0/1057892028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5844103f40 0x7f584419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 >> 172.21.15.78:0/1057892028 conn(0x7f58440ff970 msgr2=0x7f58441004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 shutdown_connections 2024-09-16T01:26:22.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:22.294+0000 7f584ae5e640 1 -- 172.21.15.78:0/1057892028 wait complete. 2024-09-16T01:26:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 -- 172.21.15.78:0/3973732889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64106a80 msgr2=0x7fea64106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 --2- 172.21.15.78:0/3973732889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64106a80 0x7fea64106e60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fea58009930 tx=0x7fea5802a420 comp rx=0 tx=0).stop 2024-09-16T01:26:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 -- 172.21.15.78:0/3973732889 shutdown_connections 2024-09-16T01:26:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 --2- 172.21.15.78:0/3973732889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64101420 0x7fea6410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 --2- 172.21.15.78:0/3973732889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64100a80 0x7fea64100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 --2- 172.21.15.78:0/3973732889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64106a80 0x7fea64106e60 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 -- 172.21.15.78:0/3973732889 >> 172.21.15.78:0/3973732889 conn(0x7fea640fc7a0 msgr2=0x7fea640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 -- 172.21.15.78:0/3973732889 shutdown_connections 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.539+0000 7fea6aba5640 1 -- 172.21.15.78:0/3973732889 wait complete. 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 Processor -- start 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 -- start start 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 0x7fea641997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 0x7fea641a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea64110e80 con 0x7fea64100a80 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fea64110d00 con 0x7fea64101420 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea6aba5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea64111000 con 0x7fea64106a80 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43444/0 (socket says 172.21.15.78:43444) 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 -- 172.21.15.78:0/1236446737 learned_addr learned my addr 172.21.15.78:0/1236446737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea69142640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 0x7fea641a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea63fff640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 0x7fea641997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 -- 172.21.15.78:0/1236446737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 msgr2=0x7fea641a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 0x7fea641a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 -- 172.21.15.78:0/1236446737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 msgr2=0x7fea641997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 0x7fea641997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea68941640 1 -- 172.21.15.78:0/1236446737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea641a4850 con 0x7fea64100a80 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea69142640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 0x7fea641a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.540+0000 7fea63fff640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 0x7fea641997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea68941640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fea58009230 tx=0x7fea58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5803f030 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea58003ad0 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea641a4a20 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58035070 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.541+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea641a4e70 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.542+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea64100ee0 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.543+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fea58042030 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.544+0000 7fea61ffb640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 0x7fea540a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.544+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7fea580f0a80 con 0x7fea64100a80 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.544+0000 7fea63fff640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 0x7fea540a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.544+0000 7fea63fff640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 0x7fea540a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea6406b2e0 tx=0x7fea50005e50 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:22.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.547+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fea580bd070 con 0x7fea64100a80 2024-09-16T01:26:22.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.749+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea6406ad80 con 0x7fea64100a80 2024-09-16T01:26:22.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.750+0000 7fea61ffb640 1 -- 172.21.15.78:0/1236446737 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7fea580bc970 con 0x7fea64100a80 2024-09-16T01:26:22.752 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:22.752 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:26:22.752 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.752+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 msgr2=0x7fea540a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.752+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 0x7fea540a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea6406b2e0 tx=0x7fea50005e50 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.752+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 msgr2=0x7fea64199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.752+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fea58009230 tx=0x7fea58004270 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 shutdown_connections 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fea540a33b0 0x7fea540a5870 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea64106a80 0x7fea641a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea64101420 0x7fea641997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 --2- 172.21.15.78:0/1236446737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea64100a80 0x7fea64199290 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:22.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 >> 172.21.15.78:0/1236446737 conn(0x7fea640fc7a0 msgr2=0x7fea640fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:22.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 shutdown_connections 2024-09-16T01:26:22.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:22.753+0000 7fea6aba5640 1 -- 172.21.15.78:0/1236446737 wait complete. 2024-09-16T01:26:23.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:23 smithi078 ceph-mon[30937]: pgmap v491: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 226 op/s 2024-09-16T01:26:23.398 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:23 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/1236446737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:23.427 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 1f1d21e5 --release --await' 2024-09-16T01:26:23.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 1f1d21e5 --release --await 2024-09-16T01:26:23.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:23 smithi193 ceph-mon[35701]: pgmap v491: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 226 op/s 2024-09-16T01:26:23.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:23 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/1236446737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:23.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:23 smithi148 ceph-mon[35030]: pgmap v491: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 226 op/s 2024-09-16T01:26:23.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:23 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/1236446737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:24.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 -- 172.21.15.78:0/3263180940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc106f20 msgr2=0x7f3efc10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 --2- 172.21.15.78:0/3263180940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc106f20 0x7f3efc10eaa0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009ca0 tx=0x7f3ef402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 -- 172.21.15.78:0/3263180940 shutdown_connections 2024-09-16T01:26:24.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 --2- 172.21.15.78:0/3263180940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc106f20 0x7f3efc10eaa0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 --2- 172.21.15.78:0/3263180940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc1027a0 0x7f3efc1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 --2- 172.21.15.78:0/3263180940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc101df0 0x7f3efc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 -- 172.21.15.78:0/3263180940 >> 172.21.15.78:0/3263180940 conn(0x7f3efc0fd8b0 msgr2=0x7f3efc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 -- 172.21.15.78:0/3263180940 shutdown_connections 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.170+0000 7f3f0252e640 1 -- 172.21.15.78:0/3263180940 wait complete. 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 Processor -- start 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 -- start start 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 0x7f3efc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc106f20 0x7f3efc19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc111fa0 con 0x7f3efc1027a0 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc111e20 con 0x7f3efc101df0 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f0252e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc112120 con 0x7f3efc106f20 2024-09-16T01:26:24.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3efb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3efb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54088/0 (socket says 172.21.15.78:54088) 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3efb7fe640 1 -- 172.21.15.78:0/2107849296 learned_addr learned my addr 172.21.15.78:0/2107849296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3f00acb640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc106f20 0x7f3efc19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.171+0000 7f3efbfff640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 0x7f3efc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 -- 172.21.15.78:0/2107849296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc106f20 msgr2=0x7f3efc19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc106f20 0x7f3efc19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 -- 172.21.15.78:0/2107849296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 msgr2=0x7f3efc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 0x7f3efc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 -- 172.21.15.78:0/2107849296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc19d3d0 con 0x7f3efc1027a0 2024-09-16T01:26:24.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efbfff640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 0x7f3efc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000d950 tx=0x7f3ef000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3ef97fa640 1 -- 172.21.15.78:0/2107849296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0017070 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc19d6c0 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3ef97fa640 1 -- 172.21.15.78:0/2107849296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef0009c30 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3ef97fa640 1 -- 172.21.15.78:0/2107849296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef000f6b0 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.172+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc1a4fa0 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.174+0000 7f3ef97fa640 1 -- 172.21.15.78:0/2107849296 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3ef00109d0 con 0x7f3efc1027a0 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.174+0000 7f3f04907740 1 Processor -- start 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.174+0000 7f3f04907740 1 -- start start 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318bb9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a318b11620 con 0x55a318bac060 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a318ab8e50 con 0x55a318bb7240 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f04907740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a318b0d5e0 con 0x55a318bac980 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54104/0 (socket says 172.21.15.78:54104) 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 -- 172.21.15.78:0/1750894450 learned_addr learned my addr 172.21.15.78:0/1750894450 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3ef97fa640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 0x7f3ee40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318bb9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 0x7f3ee40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3ef97fa640 1 -- 172.21.15.78:0/2107849296 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f3ef00cfd80 con 0x7f3efc1027a0 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 -- 172.21.15.78:0/1750894450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 msgr2=0x55a318bb6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 -- 172.21.15.78:0/1750894450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 msgr2=0x55a318bb9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318bb9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 -- 172.21.15.78:0/1750894450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a318aa9d40 con 0x55a318bac060 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.175+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4004af0 tx=0x7f3ef4003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b1919f2ad9c726 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3ee27fc640 1 -- 172.21.15.78:0/1750894450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4003e40 con 0x55a318bac060 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f00acb640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 0x7f3ee40a5870 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ef404e000 tx=0x7f3ef404c000 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3ee27fc640 1 -- 172.21.15.78:0/1750894450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef404f070 con 0x55a318bac060 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3ee27fc640 1 -- 172.21.15.78:0/1750894450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef402fd20 con 0x55a318bac060 2024-09-16T01:26:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 -- 172.21.15.78:0/1750894450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 msgr2=0x55a318bac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4004af0 tx=0x7f3ef4003a70 comp rx=0 tx=0).stop 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 -- 172.21.15.78:0/1750894450 shutdown_connections 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318bb9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 --2- 172.21.15.78:0/1750894450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bac440 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 -- 172.21.15.78:0/1750894450 >> 172.21.15.78:0/1750894450 conn(0x55a318ba85f0 msgr2=0x55a318ba89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 -- 172.21.15.78:0/1750894450 shutdown_connections 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.176+0000 7f3f04907740 1 -- 172.21.15.78:0/1750894450 wait complete. 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 -- start start 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 Processor -- start 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318bb4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55a318bb9c90 con 0x55a318bb7240 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a318a99f10 con 0x55a318bac060 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a318bb9e10 con 0x55a318bac980 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f04907740 10 client.0 _ll_register_callbacks cb 0x55a318abddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f00acb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f00acb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54120/0 (socket says 172.21.15.78:54120) 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.177+0000 7f3f00acb640 1 -- 172.21.15.78:0/1000066544 learned_addr learned my addr 172.21.15.78:0/1000066544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318bb4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 -- 172.21.15.78:0/1000066544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 msgr2=0x55a318bb4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 -- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 msgr2=0x55a318bb4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318bb4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 -- 172.21.15.78:0/1000066544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a318820580 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318bb4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f00acb640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3ef402ade0 tx=0x7f3ef402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea5a9e6a1d6c00d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef405b070 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f3ef40318e0 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef404f070 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ef4067050 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f04907740 10 client.17166 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a3188ea960 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef404f070 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.178+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3ef4030c30 con 0x55a318bb7240 2024-09-16T01:26:24.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 10 client.17166 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 10 client.17166 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 20 client.17166 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318bc8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 4 client.17166 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a318b1eb90 con 0x55a318bc6af0 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318bc8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3edb7fe640 10 client.17166 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.179+0000 7f3efbfff640 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318bc8ee0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a662b2e29a30cbd server_cookie=73382ebb0e28f17c in_seq=0 out_seq=0 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3edb7fe640 1 -- 172.21.15.78:0/1000066544 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a318b1eb90 con 0x55a318bc6af0 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3edb7fe640 10 client.17166 handle_command_reply: tid=0 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 client.17166 shutdown 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 msgr2=0x55a318bb4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3ef402ade0 tx=0x7f3ef402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 shutdown_connections 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318bc8ee0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bb7240 0x55a318bb4bd0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bb4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3edb7fe640 0 client.17166 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 --2- 172.21.15.78:0/1000066544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318bb4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 >> 172.21.15.78:0/1000066544 conn(0x55a318ba85f0 msgr2=0x55a318ba7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 shutdown_connections 2024-09-16T01:26:24.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 1 -- 172.21.15.78:0/1000066544 wait complete. 2024-09-16T01:26:24.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.180+0000 7f3f04907740 20 client.17166 trim_cache size 0 max 16384 2024-09-16T01:26:24.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 Processor -- start 2024-09-16T01:26:24.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 -- start start 2024-09-16T01:26:24.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318baeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bb7240 0x55a318bce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1b1a80 con 0x55a318bac980 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1af350 con 0x55a318bac060 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3f04907740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1ad1c0 con 0x55a318bb7240 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.198+0000 7f3efb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54128/0 (socket says 172.21.15.78:54128) 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 -- 172.21.15.78:0/4142017952 learned_addr learned my addr 172.21.15.78:0/4142017952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efbfff640 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318baeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3f00acb640 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bb7240 0x55a318bce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 -- 172.21.15.78:0/4142017952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bb7240 msgr2=0x55a318bce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bb7240 0x55a318bce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 -- 172.21.15.78:0/4142017952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 msgr2=0x55a318baeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318baeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 -- 172.21.15.78:0/4142017952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a318bac7b0 con 0x55a318bac980 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00cd190 tx=0x7f3ef001e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844ccdf98b9c89df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3ee1ffb640 1 -- 172.21.15.78:0/4142017952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0020040 con 0x55a318bac980 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3ee1ffb640 1 -- 172.21.15.78:0/4142017952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef009c370 con 0x55a318bac980 2024-09-16T01:26:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.199+0000 7f3ee1ffb640 1 -- 172.21.15.78:0/4142017952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef009b0c0 con 0x55a318bac980 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- 172.21.15.78:0/4142017952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 msgr2=0x55a318ba7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00cd190 tx=0x7f3ef001e740 comp rx=0 tx=0).stop 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- 172.21.15.78:0/4142017952 shutdown_connections 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bb7240 0x55a318bce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac980 0x55a318ba7640 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 --2- 172.21.15.78:0/4142017952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bac060 0x55a318baeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- 172.21.15.78:0/4142017952 >> 172.21.15.78:0/4142017952 conn(0x55a318ba85f0 msgr2=0x55a318ba6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- 172.21.15.78:0/4142017952 shutdown_connections 2024-09-16T01:26:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- 172.21.15.78:0/4142017952 wait complete. 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 -- start start 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.200+0000 7f3f04907740 1 Processor -- start 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bbbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318c2ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1ad810 con 0x55a318bac060 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1aec80 con 0x55a318bb7240 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1adee0 con 0x55a318bac980 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f04907740 10 client.0 _ll_register_callbacks cb 0x7f3ef40047f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54130/0 (socket says 172.21.15.78:54130) 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 -- 172.21.15.78:0/2588671466 learned_addr learned my addr 172.21.15.78:0/2588671466 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bbbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3f00acb640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318c2ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 -- 172.21.15.78:0/2588671466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 msgr2=0x55a318bbbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bbbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 -- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 msgr2=0x55a318c2ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318c2ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 -- 172.21.15.78:0/2588671466 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a318c2f040 con 0x55a318bac060 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3efbfff640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00bc40 tx=0x7f3eec016800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6bac836e69fa0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.201+0000 7f3edb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec023070 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f3eec01f650 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef009c370 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3eec02f050 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 10 client.17172 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a318c36470 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef009c370 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3eec015a00 con 0x55a318bac060 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 10 client.17172 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 10 client.17172 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 20 client.17172 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318c38be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 4 client.17172 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1f1d21e5", "release": true, "await": true}] 2024-09-16T01:26:24.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.202+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1f1d21e5", "release": true, "await": true}) -- 0x55a318b190a0 con 0x55a318bc6af0 2024-09-16T01:26:24.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.203+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318c38be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:24.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.203+0000 7f3efb7fe640 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318c38be0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b0ed772ca8b322f server_cookie=fa8ce99288454efc in_seq=0 out_seq=0 2024-09-16T01:26:24.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.203+0000 7f3edb7fe640 10 client.17172 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3edb7fe640 1 -- 172.21.15.78:0/2588671466 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55a318b190a0 con 0x55a318bc6af0 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3edb7fe640 10 client.17172 handle_command_reply: tid=0 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 client.17172 shutdown 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 msgr2=0x55a318bbb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00bc40 tx=0x7f3eec016800 comp rx=0 tx=0).stop 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 shutdown_connections 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x55a318bc6af0 0x55a318c38be0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a318bb7240 0x55a318c2ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3edb7fe640 0 client.17172 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a318bac980 0x55a318bbbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 --2- 172.21.15.78:0/2588671466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a318bac060 0x55a318bbb700 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 >> 172.21.15.78:0/2588671466 conn(0x55a318ba85f0 msgr2=0x55a318bb4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 shutdown_connections 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 1 -- 172.21.15.78:0/2588671466 wait complete. 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.263+0000 7f3f04907740 20 client.17172 trim_cache size 0 max 16384 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 60, 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": { 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "1f1d21e5": { 2024-09-16T01:26:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: "version": 60, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age_ref": 0.0, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "timeout": 90.0, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "expiration": 137.7, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "members": { 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "file:/": { 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "excluded": false, 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "state": { 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RELEASED", 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.0 2024-09-16T01:26:24.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:24.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:24.268 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 msgr2=0x7f3ee40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 0x7f3ee40a5870 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3ef404e000 tx=0x7f3ef404c000 comp rx=0 tx=0).stop 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 msgr2=0x7f3efc198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000d950 tx=0x7f3ef000de10 comp rx=0 tx=0).stop 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 shutdown_connections 2024-09-16T01:26:24.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ee40a33b0 0x7f3ee40a5870 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3efc106f20 0x7f3efc19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3efc1027a0 0x7f3efc198960 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 --2- 172.21.15.78:0/2107849296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3efc101df0 0x7f3efc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 >> 172.21.15.78:0/2107849296 conn(0x7f3efc0fd8b0 msgr2=0x7f3efc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 shutdown_connections 2024-09-16T01:26:24.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:24.264+0000 7f3f0252e640 1 -- 172.21.15.78:0/2107849296 wait complete. 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:transaction type: 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:scaling factor: 32 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:query mode: simple 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:number of clients: 100 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:number of threads: 4 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:duration: 900 s 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:number of transactions actually processed: 763430 2024-09-16T01:26:24.619 INFO:teuthology.orchestra.run.smithi078.stdout:latency average = 118.988 ms 2024-09-16T01:26:24.620 INFO:teuthology.orchestra.run.smithi078.stdout:latency stddev = 885.212 ms 2024-09-16T01:26:24.620 INFO:teuthology.orchestra.run.smithi078.stdout:tps = 840.152326 (including connections establishing) 2024-09-16T01:26:24.620 INFO:teuthology.orchestra.run.smithi078.stdout:tps = 840.154282 (excluding connections establishing) 2024-09-16T01:26:24.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-09-16T01:26:25.138 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1f1d21e5', total seconds elapsed: 21.0 2024-09-16T01:26:25.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:25 smithi078 ceph-mon[30937]: pgmap v492: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 224 op/s 2024-09-16T01:26:25.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:25 smithi193 ceph-mon[35701]: pgmap v492: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 224 op/s 2024-09-16T01:26:25.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:25 smithi148 ceph-mon[35030]: pgmap v492: 129 pgs: 129 active+clean; 3.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 3.2 MiB/s wr, 224 op/s 2024-09-16T01:26:27.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:27 smithi078 ceph-mon[30937]: pgmap v493: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-09-16T01:26:27.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:27 smithi193 ceph-mon[35701]: pgmap v493: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-09-16T01:26:27.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:27 smithi148 ceph-mon[35030]: pgmap v493: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-09-16T01:26:29.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:29 smithi078 ceph-mon[30937]: pgmap v494: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 847 KiB/s wr, 90 op/s 2024-09-16T01:26:29.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:29 smithi193 ceph-mon[35701]: pgmap v494: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 847 KiB/s wr, 90 op/s 2024-09-16T01:26:29.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:29 smithi148 ceph-mon[35030]: pgmap v494: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 847 KiB/s wr, 90 op/s 2024-09-16T01:26:31.437 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:31 smithi148 ceph-mon[35030]: pgmap v495: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-09-16T01:26:31.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:31 smithi078 ceph-mon[30937]: pgmap v495: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-09-16T01:26:31.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:31 smithi193 ceph-mon[35701]: pgmap v495: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-09-16T01:26:33.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:33 smithi078 ceph-mon[30937]: pgmap v496: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-09-16T01:26:33.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:33 smithi193 ceph-mon[35701]: pgmap v496: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-09-16T01:26:33.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:33 smithi148 ceph-mon[35030]: pgmap v496: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-09-16T01:26:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.460+0000 7f4baefd6640 1 -- 172.21.15.78:0/3304269771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba8108cf0 msgr2=0x7f4ba810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.460+0000 7f4baefd6640 1 --2- 172.21.15.78:0/3304269771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba8108cf0 0x7f4ba810cd30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009860 tx=0x7f4b9c02a360 comp rx=0 tx=0).stop 2024-09-16T01:26:35.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.467+0000 7f4baefd6640 1 -- 172.21.15.78:0/3304269771 shutdown_connections 2024-09-16T01:26:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.467+0000 7f4baefd6640 1 --2- 172.21.15.78:0/3304269771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba810d490 0x7f4ba8114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.467+0000 7f4baefd6640 1 --2- 172.21.15.78:0/3304269771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba8108cf0 0x7f4ba810cd30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.467+0000 7f4baefd6640 1 --2- 172.21.15.78:0/3304269771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba8108340 0x7f4ba8108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.467+0000 7f4baefd6640 1 -- 172.21.15.78:0/3304269771 >> 172.21.15.78:0/3304269771 conn(0x7f4ba8077a40 msgr2=0x7f4ba8079e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.472+0000 7f4baefd6640 1 -- 172.21.15.78:0/3304269771 shutdown_connections 2024-09-16T01:26:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.476+0000 7f4baefd6640 1 -- 172.21.15.78:0/3304269771 wait complete. 2024-09-16T01:26:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.476+0000 7f4baefd6640 1 Processor -- start 2024-09-16T01:26:35.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.477+0000 7f4baefd6640 1 -- start start 2024-09-16T01:26:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba8108340 0x7f4ba81a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 0x7f4ba81a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81184e0 con 0x7f4ba819e730 2024-09-16T01:26:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba8118360 con 0x7f4ba810d490 2024-09-16T01:26:35.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.478+0000 7f4baefd6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba8118660 con 0x7f4ba8108340 2024-09-16T01:26:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.481+0000 7f4bad573640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.481+0000 7f4bad573640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39670/0 (socket says 172.21.15.78:39670) 2024-09-16T01:26:35.483 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.481+0000 7f4bad573640 1 -- 172.21.15.78:0/1816939753 learned_addr learned my addr 172.21.15.78:0/1816939753 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.483+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 0x7f4ba81a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.483+0000 7f4bacd72640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba8108340 0x7f4ba81a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.487+0000 7f4bad573640 1 -- 172.21.15.78:0/1816939753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba8108340 msgr2=0x7f4ba81a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.487+0000 7f4bad573640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba8108340 0x7f4ba81a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.487+0000 7f4bad573640 1 -- 172.21.15.78:0/1816939753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 msgr2=0x7f4ba81a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.487+0000 7f4bad573640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 0x7f4ba81a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.487+0000 7f4bad573640 1 -- 172.21.15.78:0/1816939753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba819f330 con 0x7f4ba819e730 2024-09-16T01:26:35.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.491+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 0x7f4ba81a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.500+0000 7f4bad573640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000d950 tx=0x7f4ba000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.506+0000 7f4ba5ffb640 1 -- 172.21.15.78:0/1816939753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0017030 con 0x7f4ba819e730 2024-09-16T01:26:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.506+0000 7f4ba5ffb640 1 -- 172.21.15.78:0/1816939753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0004020 con 0x7f4ba819e730 2024-09-16T01:26:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.506+0000 7f4ba5ffb640 1 -- 172.21.15.78:0/1816939753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003c00 con 0x7f4ba819e730 2024-09-16T01:26:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.507+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba81ab410 con 0x7f4ba819e730 2024-09-16T01:26:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.509+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba81ab7c0 con 0x7f4ba819e730 2024-09-16T01:26:35.522 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.518+0000 7f4bb1e52740 1 Processor -- start 2024-09-16T01:26:35.523 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.521+0000 7f4ba5ffb640 1 -- 172.21.15.78:0/1816939753 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f4ba001a070 con 0x7f4ba819e730 2024-09-16T01:26:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.523+0000 7f4ba5ffb640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 0x7f4b900a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.524+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 0x7f4b900a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.526+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 0x7f4b900a5ac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02a950 tx=0x7f4b9c005760 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.527+0000 7f4bb1e52740 1 -- start start 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.527+0000 7f4ba5ffb640 1 -- 172.21.15.78:0/1816939753 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f4ba00d2170 con 0x7f4ba819e730 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbe70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562abbb3e2f0 con 0x562abbbe4cb0 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562abbae6270 con 0x562abbbda410 2024-09-16T01:26:35.529 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562abbb3a2b0 con 0x562abbbd9af0 2024-09-16T01:26:35.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:35 smithi078 ceph-mon[30937]: pgmap v497: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.3 MiB/s wr, 473 op/s 2024-09-16T01:26:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58944/0 (socket says 172.21.15.78:58944) 2024-09-16T01:26:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.528+0000 7f4bacd72640 1 -- 172.21.15.78:0/535142243 learned_addr learned my addr 172.21.15.78:0/535142243 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.530+0000 7f4bacd72640 1 -- 172.21.15.78:0/535142243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 msgr2=0x562abbbe4770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:26:35.532 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.530+0000 7f4bacd72640 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe4770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.530+0000 7f4bacd72640 1 -- 172.21.15.78:0/535142243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 msgr2=0x562abbbe70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.530+0000 7f4bacd72640 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbe70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.533 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.530+0000 7f4bacd72640 1 -- 172.21.15.78:0/535142243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562abbae5340 con 0x562abbbd9af0 2024-09-16T01:26:35.536 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.534+0000 7f4bacd72640 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800e5d0 tx=0x7f4b980095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e3998bab5d044e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.538+0000 7f4b8effd640 1 -- 172.21.15.78:0/535142243 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98004b20 con 0x562abbbd9af0 2024-09-16T01:26:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.538+0000 7f4b8effd640 1 -- 172.21.15.78:0/535142243 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b9800ddb0 con 0x562abbbd9af0 2024-09-16T01:26:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.539+0000 7f4bb1e52740 1 -- 172.21.15.78:0/535142243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 msgr2=0x562abbbd9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.539+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800e5d0 tx=0x7f4b980095c0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.543+0000 7f4bb1e52740 1 -- 172.21.15.78:0/535142243 shutdown_connections 2024-09-16T01:26:35.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.543+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbe70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.543+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.543+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/535142243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbd9ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.543+0000 7f4bb1e52740 1 -- 172.21.15.78:0/535142243 >> 172.21.15.78:0/535142243 conn(0x562abbbd6060 msgr2=0x562abbbd6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:35 smithi193 ceph-mon[35701]: pgmap v497: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.3 MiB/s wr, 473 op/s 2024-09-16T01:26:35.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.550+0000 7f4bb1e52740 1 -- 172.21.15.78:0/535142243 shutdown_connections 2024-09-16T01:26:35.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.554+0000 7f4bb1e52740 1 -- 172.21.15.78:0/535142243 wait complete. 2024-09-16T01:26:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.554+0000 7f4bb1e52740 1 -- start start 2024-09-16T01:26:35.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.554+0000 7f4bb1e52740 1 Processor -- start 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbd9af0 0x562abbbe1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x562abbbe78d0 con 0x562abbbd9af0 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562abbac7290 con 0x562abbbda410 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562abbbe7a50 con 0x562abbbe4cb0 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.560+0000 7f4bb1e52740 10 client.0 _ll_register_callbacks cb 0x562abbaf8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.561+0000 7f4bad573640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.561+0000 7f4bad573640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58956/0 (socket says 172.21.15.78:58956) 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.561+0000 7f4bad573640 1 -- 172.21.15.78:0/781068274 learned_addr learned my addr 172.21.15.78:0/781068274 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.562+0000 7f4bacd72640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbd9af0 0x562abbbe1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.564+0000 7f4bad573640 1 -- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 msgr2=0x562abbbe2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.564+0000 7f4bad573640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.564+0000 7f4bad573640 1 -- 172.21.15.78:0/781068274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbd9af0 msgr2=0x562abbbe1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.564+0000 7f4bad573640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbd9af0 0x562abbbe1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.564+0000 7f4bad573640 1 -- 172.21.15.78:0/781068274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562abb851210 con 0x562abbbe4cb0 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.565+0000 7f4bad573640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4ba009abd0 tx=0x7f4ba009ab40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d4cf94bfbdfb9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.566+0000 7f4b87fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:26:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.570+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00d3060 con 0x562abbbe4cb0 2024-09-16T01:26:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.570+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b9800ddb0 con 0x562abbbe4cb0 2024-09-16T01:26:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.570+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f4ba00041d0 con 0x562abbbe4cb0 2024-09-16T01:26:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.575+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ba009aed0 con 0x562abbbe4cb0 2024-09-16T01:26:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.576+0000 7f4bb1e52740 10 client.26449 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.576+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562abb91d620 con 0x562abbbe4cb0 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b9800ddb0 con 0x562abbbe4cb0 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4ba009a320 con 0x562abbbe4cb0 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 10 client.26449 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 10 client.26449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 20 client.26449 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbbf4c10 0x562abbbf7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 4 client.26449 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.581+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562abbb4ba20 con 0x562abbbf4c10 2024-09-16T01:26:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.582+0000 7f4bacd72640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbbf4c10 0x562abbbf7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.583+0000 7f4bacd72640 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbbf4c10 0x562abbbf7000 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62da05b0242b9f5 server_cookie=c75d58d0e571fc67 in_seq=0 out_seq=0 2024-09-16T01:26:35.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.585+0000 7f4b87fff640 10 client.26449 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.588+0000 7f4b87fff640 1 -- 172.21.15.78:0/781068274 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562abbb4ba20 con 0x562abbbf4c10 2024-09-16T01:26:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.588+0000 7f4b87fff640 10 client.26449 handle_command_reply: tid=0 2024-09-16T01:26:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.588+0000 7f4bb1e52740 1 client.26449 shutdown 2024-09-16T01:26:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.594+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 msgr2=0x562abbbe2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.594+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4ba009abd0 tx=0x7f4ba009ab40 comp rx=0 tx=0).stop 2024-09-16T01:26:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 shutdown_connections 2024-09-16T01:26:35.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbbf4c10 0x562abbbf7000 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbe4cb0 0x562abbbe2580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbda410 0x562abbbe2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/781068274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbd9af0 0x562abbbe1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.596+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 >> 172.21.15.78:0/781068274 conn(0x562abbbd6060 msgr2=0x562abbbdbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.597+0000 7f4b87fff640 0 client.26449 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.601+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 shutdown_connections 2024-09-16T01:26:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.602+0000 7f4bb1e52740 1 -- 172.21.15.78:0/781068274 wait complete. 2024-09-16T01:26:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.605+0000 7f4bb1e52740 20 client.26449 trim_cache size 0 max 16384 2024-09-16T01:26:35.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:35 smithi148 ceph-mon[35030]: pgmap v497: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.3 MiB/s wr, 473 op/s 2024-09-16T01:26:35.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.641+0000 7f4bb1e52740 1 Processor -- start 2024-09-16T01:26:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.641+0000 7f4bb1e52740 1 -- start start 2024-09-16T01:26:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.641+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.641+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbda410 0x562abbbdbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.78:58958/0 (socket says 172.21.15.78:58958) 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bacd72640 1 -- 172.21.15.78:0/67947673 learned_addr learned my addr 172.21.15.78:0/67947673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 0x562abbbfc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b80f0 con 0x562abbbda410 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b59c0 con 0x562abbbe4cb0 2024-09-16T01:26:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b3830 con 0x562abbbd9af0 2024-09-16T01:26:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4bad573640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 0x562abbbfc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.642+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbda410 0x562abbbdbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 -- 172.21.15.78:0/67947673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 msgr2=0x562abbbe23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.646 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 -- 172.21.15.78:0/67947673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbda410 msgr2=0x562abbbdbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbda410 0x562abbbdbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 -- 172.21.15.78:0/67947673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562abbbd5e90 con 0x562abbbe4cb0 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.643+0000 7f4bad573640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 0x562abbbfc880 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0099180 tx=0x7f4ba00993e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08d6643c437d27b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.644+0000 7f4bacd72640 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.644+0000 7f4b8e7fc640 1 -- 172.21.15.78:0/67947673 <== mon.2 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba009f280 con 0x562abbbe4cb0 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.644+0000 7f4b8e7fc640 1 -- 172.21.15.78:0/67947673 <== mon.2 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba00a3b40 con 0x562abbbe4cb0 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.645+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 msgr2=0x562abbbfc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.645+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 0x562abbbfc880 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0099180 tx=0x7f4ba00993e0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.649+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 shutdown_connections 2024-09-16T01:26:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.649+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbe4cb0 0x562abbbfc880 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.649+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbda410 0x562abbbdbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.649+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/67947673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbd9af0 0x562abbbe23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.649+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 >> 172.21.15.78:0/67947673 conn(0x562abbbd6060 msgr2=0x562abbbdb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.652+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 shutdown_connections 2024-09-16T01:26:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.652+0000 7f4bb1e52740 1 -- 172.21.15.78:0/67947673 wait complete. 2024-09-16T01:26:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 -- start start 2024-09-16T01:26:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 Processor -- start 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 0x562abbbed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 0x562abbbde930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbdef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b3e80 con 0x562abbbe4cb0 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b52f0 con 0x562abbbd9af0 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba81b4550 con 0x562abbbda410 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.653+0000 7f4bb1e52740 10 client.0 _ll_register_callbacks cb 0x7f4ba00d1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.654+0000 7f4ba7fff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 0x562abbbde930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.654+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 0x562abbbed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.654+0000 7f4bacd72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 0x562abbbed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.78:57946/0 (socket says 172.21.15.78:57946) 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.654+0000 7f4bacd72640 1 -- 172.21.15.78:0/1232771842 learned_addr learned my addr 172.21.15.78:0/1232771842 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4bad573640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbdef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4ba7fff640 1 -- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 msgr2=0x562abbbed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 0x562abbbed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4ba7fff640 1 -- 172.21.15.78:0/1232771842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 msgr2=0x562abbbdef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbdef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.655+0000 7f4ba7fff640 1 -- 172.21.15.78:0/1232771842 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562abbc6f860 con 0x562abbbda410 2024-09-16T01:26:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.656+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 0x562abbbde930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c035040 tx=0x7f4b9c02a360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8f5561e6222e00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.656+0000 7f4b87fff640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-09-16T01:26:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.660+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c04f060 con 0x562abbbda410 2024-09-16T01:26:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.660+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f4b9c004450 con 0x562abbbda410 2024-09-16T01:26:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.661+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ba00a3b40 con 0x562abbbda410 2024-09-16T01:26:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.662+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4b9c037040 con 0x562abbbda410 2024-09-16T01:26:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.666+0000 7f4bb1e52740 10 client.26452 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.666+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562abbc708f0 con 0x562abbbda410 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.668+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba00a3b40 con 0x562abbbda410 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.668+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4b9c03a030 con 0x562abbbda410 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 10 client.26452 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 10 client.26452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 20 client.26452 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbc71050 0x562abbc73470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 4 client.26452 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:26:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.670+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562abbb45d70 con 0x562abbc71050 2024-09-16T01:26:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.671+0000 7f4bacd72640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbc71050 0x562abbc73470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:35.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.678+0000 7f4b87fff640 10 client.26452 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.678+0000 7f4bacd72640 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbc71050 0x562abbc73470 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a69cd8bc6e709d server_cookie=4d6e2cb1c2b915d4 in_seq=0 out_seq=0 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4b87fff640 1 -- 172.21.15.78:0/1232771842 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562abbb45d70 con 0x562abbc71050 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4b87fff640 10 client.26452 handle_command_reply: tid=0 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 client.26452 shutdown 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 msgr2=0x562abbbde930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 0x562abbbde930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c035040 tx=0x7f4b9c02a360 comp rx=0 tx=0).stop 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 shutdown_connections 2024-09-16T01:26:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x562abbc71050 0x562abbc73470 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562abbbe4cb0 0x562abbbdef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562abbbda410 0x562abbbde930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 --2- 172.21.15.78:0/1232771842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562abbbd9af0 0x562abbbed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 >> 172.21.15.78:0/1232771842 conn(0x562abbbd6060 msgr2=0x562abbbdbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.932+0000 7f4b87fff640 0 client.26452 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.933+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 shutdown_connections 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.933+0000 7f4bb1e52740 1 -- 172.21.15.78:0/1232771842 wait complete. 2024-09-16T01:26:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.933+0000 7f4bb1e52740 20 client.26452 trim_cache size 0 max 16384 2024-09-16T01:26:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2024-09-16T01:26:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2024-09-16T01:26:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2024-09-16T01:26:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 msgr2=0x7f4b900a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 0x7f4b900a5ac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c02a950 tx=0x7f4b9c005760 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 msgr2=0x7f4ba819ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000d950 tx=0x7f4ba000de10 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 shutdown_connections 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f4b900a3600 0x7f4b900a5ac0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba819e730 0x7f4ba819ebb0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ba810d490 0x7f4ba81a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 --2- 172.21.15.78:0/1816939753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ba8108340 0x7f4ba81a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 >> 172.21.15.78:0/1816939753 conn(0x7f4ba8077a40 msgr2=0x7f4ba8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 shutdown_connections 2024-09-16T01:26:35.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2024-09-16T01:26:35.934+0000 7f4baefd6640 1 -- 172.21.15.78:0/1816939753 wait complete. 2024-09-16T01:26:36.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/: 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 data 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data: 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:total 36 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 . 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 .. 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 base 2024-09-16T01:26:36.735 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Sep 16 01:11 current_logfiles 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Sep 16 01:11 global 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 log 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_commit_ts 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_dynshmem 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Sep 16 01:11 pg_hba.conf 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Sep 16 01:11 pg_ident.conf 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 pg_logical 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 pg_multixact 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_notify 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_replslot 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_serial 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_snapshots 2024-09-16T01:26:36.736 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 pg_stat 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:26 pg_stat_tmp 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:26 pg_subtrans 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_tblspc 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pg_twophase 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 59 Sep 16 01:26 pg_wal 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 pg_xact 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Sep 16 01:11 postgresql.auto.conf 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28074 Sep 16 01:11 postgresql.conf 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Sep 16 01:11 postmaster.opts 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.737 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base: 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 . 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 1 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 13493 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Sep 16 01:17 13494 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 pgsql_tmp 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/1: 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:total 7777 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 . 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 .. 2024-09-16T01:26:36.738 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:11 1259 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:26:36.739 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:26:36.740 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:26:36.741 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:26:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:26:36.743 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:26:36.744 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:26:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 16 01:11 2619 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2619_fsm 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:26:36.746 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:26:36.747 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:26:36.748 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2696 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:26:36.749 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:26:36.750 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:26:36.751 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:26:36.752 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:26:36.753 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:26:36.754 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:26:36.755 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:26:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:26:36.757 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:26:36.758 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:26:36.759 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:26:36.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:26:36.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:26:36.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:26:36.760 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:26:36.761 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:26:36.762 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:total 7777 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 16 01:11 . 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 .. 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:26:36.763 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:11 1259 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:26:36.764 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:26:36.765 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:26:36.766 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:26:36.767 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:26:36.768 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:26:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 16 01:11 2619 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2619_fsm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:26:36.770 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:26:36.771 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:26:36.772 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:26:36.773 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2696 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:26:36.774 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:26:36.775 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:26:36.776 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:26:36.777 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:26:36.778 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:26:36.779 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:26:36.780 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:26:36.781 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:26:36.782 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:26:36.783 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:26:36.784 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:26:36.788 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:26:36.788 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:26:36.788 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:26:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:total 550174 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Sep 16 01:17 . 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 .. 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 112 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 113 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 1247 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1247_fsm 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1247_vm 2024-09-16T01:26:36.790 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 16 01:11 1249 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1249_fsm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1249_vm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 16 01:11 1255 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1255_fsm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1255_vm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 16 01:26 1259 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1259_fsm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1259_vm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 16 01:11 13344 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13344_fsm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13344_vm 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13346 2024-09-16T01:26:36.791 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13348 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13349_fsm 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13349_vm 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13351 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13353 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13354_fsm 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13354_vm 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13356 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13358 2024-09-16T01:26:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 13359_fsm 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13359_vm 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 13361 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 13363 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1417 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 1418 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 440066048 Sep 16 01:26 16396 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Sep 16 01:26 16396_fsm 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:21 16396_vm 2024-09-16T01:26:36.793 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 802816 Sep 16 01:26 16397 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:26 16397_fsm 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:26 16397_vm 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1900544 Sep 16 01:26 16399 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:26 16399_fsm 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:22 16399_vm 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:26 16400 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:26 16402 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Sep 16 01:26 16404 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40230912 Sep 16 01:26 16406 2024-09-16T01:26:36.794 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:26 16406_fsm 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:24 16406_vm 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 174 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 175 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2187 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2224 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2328 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2336 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2337 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2579 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2600 2024-09-16T01:26:36.795 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2600_fsm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2600_vm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2601_fsm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2601_vm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2602 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2602_fsm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2602_vm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2603 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2603_fsm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2603_vm 2024-09-16T01:26:36.796 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2604 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2605 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2605_fsm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2605_vm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2606_fsm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2606_vm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2607 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2607_fsm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2607_vm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 16 01:11 2608 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2608_fsm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2608_vm 2024-09-16T01:26:36.797 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 16 01:11 2609 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2609_fsm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2609_vm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2610 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2610_fsm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2610_vm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2611 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2612_fsm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2612_vm 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2613 2024-09-16T01:26:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2615_fsm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2615_vm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2616 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2616_fsm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2616_vm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 16 01:11 2617 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2617_fsm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2617_vm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 16 01:11 2618 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2618_fsm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2618_vm 2024-09-16T01:26:36.799 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Sep 16 01:26 2619 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:14 2619_fsm 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2619_vm 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2620 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2650 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2651 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2652 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2653 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2654 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2655 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2656 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2657 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 16 01:11 2658 2024-09-16T01:26:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 16 01:11 2659 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2660 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2661 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2662 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2663 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2664 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2665 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2666 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2667 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2668 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2669 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2670 2024-09-16T01:26:36.801 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 16 01:11 2673 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 16 01:11 2674 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:11 2675 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2678 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2679 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2680 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2681 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2682 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2683 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2684 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2685 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2686 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2687 2024-09-16T01:26:36.802 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2688 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 16 01:11 2689 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 16 01:11 2690 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 16 01:11 2691 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2692 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2693 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:15 2696 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2699 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2701 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2702 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2703 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 2704 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2753 2024-09-16T01:26:36.803 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2753_fsm 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2753_vm 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2754 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2755 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2756 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 2757 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2830 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2831 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2832 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2833 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2834 2024-09-16T01:26:36.804 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2835 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2836 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2837 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 16 01:11 2838 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2838_fsm 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2838_vm 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2839 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2840_fsm 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2840_vm 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2841 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2995 2024-09-16T01:26:36.805 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2996 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3079_fsm 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3079_vm 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3080 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3081 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3085 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3118 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3119 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 16 01:11 3164 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3256 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3257 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3258 2024-09-16T01:26:36.806 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3350 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3351 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3379 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3380 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3381 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3394 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3394_fsm 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3394_vm 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3395 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3429 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3430 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3431 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3433 2024-09-16T01:26:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3439 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3440 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 16 01:11 3455 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 16 01:11 3456 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3456_fsm 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3456_vm 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3466 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3467 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3468 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3501 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3502 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3503 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3534 2024-09-16T01:26:36.808 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3541_fsm 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3541_vm 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3542 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3574 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3575 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3576 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3596 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3597 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3598 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3599 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3600_fsm 2024-09-16T01:26:36.809 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3600_vm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3601_fsm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3601_vm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3602_fsm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3602_vm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3603_fsm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3603_vm 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3604 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3605 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3606 2024-09-16T01:26:36.810 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3607 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3608 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 16 01:11 3609 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3712 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 3764_fsm 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3764_vm 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3766 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 3767 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3997 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4143 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4144 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4145 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4146 2024-09-16T01:26:36.811 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4147 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4148 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4149 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4150 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4151 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4152 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4153 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4154 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4155 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4156 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4157 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4158 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4159 2024-09-16T01:26:36.812 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4160 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4161 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4162 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4163 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4164 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4165 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4166 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4167 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4168 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4169 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4170 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4171 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4172 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4173 2024-09-16T01:26:36.813 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4174 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 5002 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 548 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 549 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6102 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6104 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6106 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6110 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6111 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6112 2024-09-16T01:26:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6113 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6117 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 826 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 827 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 828 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Sep 16 01:11 pg_internal.init 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:11 PG_VERSION 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 16 01:11 .. 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/global: 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:total 560 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Sep 16 01:11 . 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1213 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1213_fsm 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1213_vm 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1214 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1214_fsm 2024-09-16T01:26:36.816 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1214_vm 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 1232 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 1233 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1260 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1260_fsm 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1260_vm 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1261 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1261_fsm 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1261_vm 2024-09-16T01:26:36.817 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1262 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 1262_fsm 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 1262_vm 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2396 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 16 01:11 2396_fsm 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2396_vm 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2397 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2671 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2672 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2676 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2677 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2694 2024-09-16T01:26:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2695 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2697 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 16 01:11 2698 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2846 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2847 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2964 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2965 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 2966 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 2967 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 3592 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 3593 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4060 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4061 2024-09-16T01:26:36.819 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4175 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4176 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4177 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4178 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4181 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4182 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4183 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4184 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 4185 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 4186 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6000 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6001 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6002 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 6100 2024-09-16T01:26:36.820 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6114 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 6115 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:26 pg_control 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 16 01:11 pg_filenode.map 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Sep 16 01:11 pg_internal.init 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/log: 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:total 2 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1461 Sep 16 01:26 postgresql-Mon.log 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-09-16T01:26:36.821 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:total 1 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 . 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.822 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 mappings 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Sep 16 01:26 replorigin_checkpoint 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 snapshots 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 .. 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 .. 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.823 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 . 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 members 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 offsets 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 .. 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:11 0000 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.824 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:total 8 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 16 01:11 .. 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 16 01:16 0000 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-09-16T01:26:36.825 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.827 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:total 15 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 16 01:26 . 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Sep 16 01:26 db_0.stat 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Sep 16 01:26 db_13494.stat 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Sep 16 01:26 global.stat 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:26 . 2024-09-16T01:26:36.828 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:total 176 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:26 . 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Sep 16 01:26 000B 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.829 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:total 950272 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 59 Sep 16 01:26 . 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:26 000000010000000000000063 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 000000010000000000000064 2024-09-16T01:26:36.830 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 000000010000000000000065 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000066 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:18 000000010000000000000067 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000068 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 000000010000000000000069 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000006A 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 00000001000000000000006B 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:19 00000001000000000000006C 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:20 00000001000000000000006D 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000006E 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000006F 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:18 000000010000000000000070 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 000000010000000000000071 2024-09-16T01:26:36.831 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:18 000000010000000000000072 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000073 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000074 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000075 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 000000010000000000000076 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000077 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:20 000000010000000000000078 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000079 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000007A 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000007B 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 00000001000000000000007C 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000007D 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:17 00000001000000000000007E 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 00000001000000000000007F 2024-09-16T01:26:36.832 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:16 000000010000000000000080 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 000000010000000000000081 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 000000010000000000000082 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000083 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 000000010000000000000084 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 000000010000000000000085 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 000000010000000000000086 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000087 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:25 000000010000000000000088 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:24 000000010000000000000089 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 00000001000000000000008A 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:25 00000001000000000000008B 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 00000001000000000000008C 2024-09-16T01:26:36.833 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 00000001000000000000008D 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 00000001000000000000008E 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 00000001000000000000008F 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000090 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:23 000000010000000000000091 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:24 000000010000000000000092 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 000000010000000000000093 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 000000010000000000000094 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000095 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000096 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:25 000000010000000000000097 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:22 000000010000000000000098 2024-09-16T01:26:36.834 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 000000010000000000000099 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 00000001000000000000009A 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:21 00000001000000000000009B 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 16 01:24 00000001000000000000009C 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 archive_status 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:total 0 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 16 01:11 . 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 59 Sep 16 01:26 .. 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:total 192 2024-09-16T01:26:36.835 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 16 01:11 . 2024-09-16T01:26:36.836 INFO:teuthology.orchestra.run.smithi078.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 16 01:26 .. 2024-09-16T01:26:36.836 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 16 01:26 0000 2024-09-16T01:26:36.836 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-16T01:26:36.847 INFO:tasks.quiescer:joining Quiescers 2024-09-16T01:26:36.847 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-16T01:26:36.847 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:26:37.105 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:37.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:37 smithi078 ceph-mon[30937]: pgmap v498: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-09-16T01:26:37.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:37 smithi193 ceph-mon[35701]: pgmap v498: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-09-16T01:26:37.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 -- 172.21.15.78:0/2967734916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301036c0 msgr2=0x7efe3010bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:37.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 --2- 172.21.15.78:0/2967734916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301036c0 0x7efe3010bbb0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7efe18009900 tx=0x7efe1802a480 comp rx=0 tx=0).stop 2024-09-16T01:26:37.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 -- 172.21.15.78:0/2967734916 shutdown_connections 2024-09-16T01:26:37.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 --2- 172.21.15.78:0/2967734916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 0x7efe3010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 --2- 172.21.15.78:0/2967734916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe301036c0 0x7efe3010bbb0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 --2- 172.21.15.78:0/2967734916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe30102da0 0x7efe30103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 -- 172.21.15.78:0/2967734916 >> 172.21.15.78:0/2967734916 conn(0x7efe300fc7a0 msgr2=0x7efe300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 -- 172.21.15.78:0/2967734916 shutdown_connections 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.603+0000 7efe34d6e640 1 -- 172.21.15.78:0/2967734916 wait complete. 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 Processor -- start 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 -- start start 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe301036c0 0x7efe301989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 0x7efe3019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7efe30111f60 con 0x7efe30102da0 2024-09-16T01:26:37.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efe30111de0 con 0x7efe3010c180 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe34d6e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe301120e0 con 0x7efe301036c0 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe2e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe2e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39708/0 (socket says 172.21.15.78:39708) 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.604+0000 7efe2e59c640 1 -- 172.21.15.78:0/176032088 learned_addr learned my addr 172.21.15.78:0/176032088 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2ed9d640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 0x7efe3019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe27fff640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe301036c0 0x7efe301989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 -- 172.21.15.78:0/176032088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe301036c0 msgr2=0x7efe301989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe301036c0 0x7efe301989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 -- 172.21.15.78:0/176032088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 msgr2=0x7efe3019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 0x7efe3019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 -- 172.21.15.78:0/176032088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3019d450 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe2e59c640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d770 tx=0x7efe2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20017070 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe20005500 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3019d740 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20004e00 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.605+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe301a5020 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.606+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe30103180 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.607+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7efe20004fa0 con 0x7efe30102da0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.608+0000 7efe277fe640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 0x7efe1c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.608+0000 7efe27fff640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 0x7efe1c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:37.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.608+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7efe200cfa30 con 0x7efe30102da0 2024-09-16T01:26:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.608+0000 7efe27fff640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 0x7efe1c0a58f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efe18009230 tx=0x7efe180058d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:37.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.612+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efe2009b9d0 con 0x7efe30102da0 2024-09-16T01:26:37.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:37 smithi148 ceph-mon[35030]: pgmap v498: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-09-16T01:26:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.813+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe30107220 con 0x7efe30102da0 2024-09-16T01:26:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.814+0000 7efe277fe640 1 -- 172.21.15.78:0/176032088 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7efe2009fcd0 con 0x7efe30102da0 2024-09-16T01:26:37.815 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:37.815 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:26:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 msgr2=0x7efe1c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 0x7efe1c0a58f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efe18009230 tx=0x7efe180058d0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 msgr2=0x7efe301984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d770 tx=0x7efe2000dc30 comp rx=0 tx=0).stop 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 shutdown_connections 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7efe1c0a3430 0x7efe1c0a58f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe3010c180 0x7efe3019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe301036c0 0x7efe301989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 --2- 172.21.15.78:0/176032088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe30102da0 0x7efe301984a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 >> 172.21.15.78:0/176032088 conn(0x7efe300fc7a0 msgr2=0x7efe300fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 shutdown_connections 2024-09-16T01:26:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:37.816+0000 7efe34d6e640 1 -- 172.21.15.78:0/176032088 wait complete. 2024-09-16T01:26:38.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:38 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/176032088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:38.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:38 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/176032088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:38.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:38 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/176032088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:26:38.661 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --cancel --all' 2024-09-16T01:26:38.661 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --cancel --all 2024-09-16T01:26:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:39.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:39 smithi078 ceph-mon[30937]: pgmap v499: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-09-16T01:26:39.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.407+0000 7f073b983640 1 -- 172.21.15.78:0/892921085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734101420 msgr2=0x7f073410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.407+0000 7f073b983640 1 --2- 172.21.15.78:0/892921085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734101420 0x7f073410d950 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f072c009930 tx=0x7f072c02a420 comp rx=0 tx=0).stop 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.407+0000 7f073b983640 1 -- 172.21.15.78:0/892921085 shutdown_connections 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.407+0000 7f073b983640 1 --2- 172.21.15.78:0/892921085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734101420 0x7f073410d950 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.407+0000 7f073b983640 1 --2- 172.21.15.78:0/892921085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734100a80 0x7f0734100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 --2- 172.21.15.78:0/892921085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 0x7f0734106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- 172.21.15.78:0/892921085 >> 172.21.15.78:0/892921085 conn(0x7f07340fc7a0 msgr2=0x7f07340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- 172.21.15.78:0/892921085 shutdown_connections 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- 172.21.15.78:0/892921085 wait complete. 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 Processor -- start 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- start start 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 0x7f0734198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 0x7f073419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0734110ea0 con 0x7f0734100a80 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0734110d20 con 0x7f0734101420 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.408+0000 7f073b983640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0734111020 con 0x7f0734106a80 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39736/0 (socket says 172.21.15.78:39736) 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 -- 172.21.15.78:0/3772168089 learned_addr learned my addr 172.21.15.78:0/3772168089 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f0738f1e640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 0x7f0734198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f0739f20640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 0x7f073419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 -- 172.21.15.78:0/3772168089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 msgr2=0x7f073419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 0x7f073419cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 -- 172.21.15.78:0/3772168089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 msgr2=0x7f0734198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 0x7f0734198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 -- 172.21.15.78:0/3772168089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073419d380 con 0x7f0734100a80 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f0738f1e640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 0x7f0734198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073971f640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f072400ba50 tx=0x7f072400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f07327fc640 1 -- 172.21.15.78:0/3772168089 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724019070 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f07327fc640 1 -- 172.21.15.78:0/3772168089 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f072400cbe0 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073419d670 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f07327fc640 1 -- 172.21.15.78:0/3772168089 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240076e0 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.409+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07341a4f50 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f07327fc640 1 -- 172.21.15.78:0/3772168089 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f072401c070 con 0x7f0734100a80 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 Processor -- start 2024-09-16T01:26:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 -- start start 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e5dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555a50dc2200 con 0x555a50e5e200 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555a50d68dc0 con 0x555a50e6cff0 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073dd5c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a50dbe1c0 con 0x555a50e5d8e0 2024-09-16T01:26:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39750/0 (socket says 172.21.15.78:39750) 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f073971f640 1 -- 172.21.15.78:0/2576610708 learned_addr learned my addr 172.21.15.78:0/2576610708 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f0739f20640 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e5dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.412+0000 7f0738f1e640 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f07327fc640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 0x7f071c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 -- 172.21.15.78:0/2576610708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 msgr2=0x555a50e5dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e5dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 -- 172.21.15.78:0/2576610708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 msgr2=0x555a50e6f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 -- 172.21.15.78:0/2576610708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a50b33620 con 0x555a50e5e200 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f0739f20640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 0x7f071c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f07327fc640 1 -- 172.21.15.78:0/3772168089 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f07240cf730 con 0x7f0734100a80 2024-09-16T01:26:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f073971f640 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f07240cca80 tx=0x7f07240660c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f843fceb0e172db9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f071b7fe640 1 -- 172.21.15.78:0/2576610708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072409b210 con 0x555a50e5e200 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f0739f20640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 0x7f071c0a5870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f072c005ab0 tx=0x7f072c009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f071b7fe640 1 -- 172.21.15.78:0/2576610708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724023050 con 0x555a50e5e200 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.413+0000 7f071b7fe640 1 -- 172.21.15.78:0/2576610708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072409b030 con 0x555a50e5e200 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- 172.21.15.78:0/2576610708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 msgr2=0x555a50e6cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f07240cca80 tx=0x7f07240660c0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- 172.21.15.78:0/2576610708 shutdown_connections 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5e200 0x555a50e6cab0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 --2- 172.21.15.78:0/2576610708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e5dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- 172.21.15.78:0/2576610708 >> 172.21.15.78:0/2576610708 conn(0x555a50e59f90 msgr2=0x555a50e5a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- 172.21.15.78:0/2576610708 shutdown_connections 2024-09-16T01:26:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- 172.21.15.78:0/2576610708 wait complete. 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.414+0000 7f073dd5c740 1 -- start start 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 Processor -- start 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5e200 0x555a50e69bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x555a50e6fbc0 con 0x555a50e5d8e0 2024-09-16T01:26:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555a50d5cc50 con 0x555a50e6cff0 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a50e6fd40 con 0x555a50e5e200 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073dd5c740 10 client.0 _ll_register_callbacks cb 0x555a50d6eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39760/0 (socket says 172.21.15.78:39760) 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 -- 172.21.15.78:0/3459678121 learned_addr learned my addr 172.21.15.78:0/3459678121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 -- 172.21.15.78:0/3459678121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5e200 msgr2=0x555a50e69bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f0738f1e640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5e200 0x555a50e69bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5e200 0x555a50e69bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 -- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 msgr2=0x555a50e6a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 -- 172.21.15.78:0/3459678121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a50e6a8f0 con 0x555a50e5d8e0 2024-09-16T01:26:39.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f073971f640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f07240bb940 tx=0x7f07240ccae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e05d6124a3d11f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.415+0000 7f0718ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072409b030 con 0x555a50e5d8e0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f07240af8b0 con 0x555a50e5d8e0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0724023050 con 0x555a50e5d8e0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f072409b210 con 0x555a50e5d8e0 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 10 client.17205 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a50e7b750 con 0x555a50e5d8e0 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724023050 con 0x555a50e5d8e0 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f0724097d40 con 0x555a50e5d8e0 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 10 client.17205 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 10 client.17205 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 20 client.17205 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50e7caa0 0x555a50e7ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 4 client.17205 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:26:39.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a50dcf770 con 0x555a50e7caa0 2024-09-16T01:26:39.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.416+0000 7f0738f1e640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50e7caa0 0x555a50e7ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.417+0000 7f0718ff9640 10 client.17205 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:39.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.417+0000 7f0738f1e640 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50e7caa0 0x555a50e7ee90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a3d1b14da97a17 server_cookie=39831e94388d92b8 in_seq=0 out_seq=0 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f0718ff9640 1 -- 172.21.15.78:0/3459678121 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555a50dcf770 con 0x555a50e7caa0 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f0718ff9640 10 client.17205 handle_command_reply: tid=0 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 client.17205 shutdown 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 msgr2=0x555a50e6c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f07240bb940 tx=0x7f07240ccae0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 shutdown_connections 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50e7caa0 0x555a50e7ee90 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e6cff0 0x555a50e6a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f0718ff9640 0 client.17205 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5e200 0x555a50e69bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 --2- 172.21.15.78:0/3459678121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c900 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 >> 172.21.15.78:0/3459678121 conn(0x555a50e59f90 msgr2=0x555a50e591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 shutdown_connections 2024-09-16T01:26:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 1 -- 172.21.15.78:0/3459678121 wait complete. 2024-09-16T01:26:39.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.418+0000 7f073dd5c740 20 client.17205 trim_cache size 0 max 16384 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 Processor -- start 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 -- start start 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e84460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e6cff0 0x555a50e849a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f07341af810 con 0x555a50e5d8e0 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f07341ad0e0 con 0x555a50e5e200 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073dd5c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07341aaf50 con 0x555a50e6cff0 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39772/0 (socket says 172.21.15.78:39772) 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 -- 172.21.15.78:0/91259875 learned_addr learned my addr 172.21.15.78:0/91259875 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f0739f20640 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e6cff0 0x555a50e849a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f0738f1e640 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e84460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 -- 172.21.15.78:0/91259875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e6cff0 msgr2=0x555a50e849a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e6cff0 0x555a50e849a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 -- 172.21.15.78:0/91259875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 msgr2=0x555a50e84460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e84460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 -- 172.21.15.78:0/91259875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a50e59dc0 con 0x555a50e5d8e0 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.435+0000 7f073971f640 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0724004a80 tx=0x7f0724014d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37caf0b2f9d5e945 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f071affd640 1 -- 172.21.15.78:0/91259875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072409b030 con 0x555a50e5d8e0 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f071affd640 1 -- 172.21.15.78:0/91259875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07240ce070 con 0x555a50e5d8e0 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f071affd640 1 -- 172.21.15.78:0/91259875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240af830 con 0x555a50e5d8e0 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 -- 172.21.15.78:0/91259875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 msgr2=0x555a50e6c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0724004a80 tx=0x7f0724014d60 comp rx=0 tx=0).stop 2024-09-16T01:26:39.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 -- 172.21.15.78:0/91259875 shutdown_connections 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e6cff0 0x555a50e849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e84460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 --2- 172.21.15.78:0/91259875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e5d8e0 0x555a50e6c420 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 -- 172.21.15.78:0/91259875 >> 172.21.15.78:0/91259875 conn(0x555a50e59f90 msgr2=0x555a50e591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 -- 172.21.15.78:0/91259875 shutdown_connections 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.436+0000 7f073dd5c740 1 -- 172.21.15.78:0/91259875 wait complete. 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 -- start start 2024-09-16T01:26:39.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 Processor -- start 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e75890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e70bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f07341ab5a0 con 0x555a50e6cff0 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f07341aca10 con 0x555a50e5e200 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07341abc70 con 0x555a50e5d8e0 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073dd5c740 10 client.0 _ll_register_callbacks cb 0x7f0724097a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:26:39.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f0739f20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f0739f20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39776/0 (socket says 172.21.15.78:39776) 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f0739f20640 1 -- 172.21.15.78:0/1448217771 learned_addr learned my addr 172.21.15.78:0/1448217771 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f073971f640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e75890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.437+0000 7f0738f1e640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e70bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 -- 172.21.15.78:0/1448217771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 msgr2=0x555a50e75890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e75890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 -- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 msgr2=0x555a50e70bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e70bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 -- 172.21.15.78:0/1448217771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a50ee6430 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0738f1e640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e70bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0739f20640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f072c009930 tx=0x7f072c0314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc0ffe8e8bfeb20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c04e070 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f072c03d450 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07240ce070 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f072c05a050 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 10 client.17211 fetch_fsmap learned FSMap version 31 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a50ee75d0 con 0x555a50e6cff0 2024-09-16T01:26:39.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07240ce070 con 0x555a50e6cff0 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f072c04a730 con 0x555a50e6cff0 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 10 client.17211 fetch_fsmap finished waiting for FSMap version 31 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 10 client.17211 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 20 client.17211 populate_metadata read hostname 'smithi078' 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.438+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50ee7d50 0x555a50eea170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f073dd5c740 4 client.17211 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 --> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x555a50dc9c80 con 0x555a50ee7d50 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f073971f640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50ee7d50 0x555a50eea170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f0718ff9640 10 client.17211 ms_handle_connect on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:39.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f073971f640 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50ee7d50 0x555a50eea170 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72f6e6c313763b89 server_cookie=3104bcc817f99f64 in_seq=0 out_seq=0 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f0718ff9640 1 -- 172.21.15.78:0/1448217771 <== mds.0 v2:172.21.15.148:6838/3106047695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x555a50dc9c80 con 0x555a50ee7d50 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f0718ff9640 10 client.17211 handle_command_reply: tid=0 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.439+0000 7f073dd5c740 1 client.17211 shutdown 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 msgr2=0x555a50e71350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f072c009930 tx=0x7f072c0314b0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 shutdown_connections 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:6838/3106047695,v1:172.21.15.148:6839/3106047695] conn(0x555a50ee7d50 0x555a50eea170 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555a50e6cff0 0x555a50e71350 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555a50e5e200 0x555a50e70bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f0718ff9640 0 client.17211 ms_handle_reset on v2:172.21.15.148:6838/3106047695 2024-09-16T01:26:39.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 --2- 172.21.15.78:0/1448217771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a50e5d8e0 0x555a50e75890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 >> 172.21.15.78:0/1448217771 conn(0x555a50e59f90 msgr2=0x555a50e5f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 shutdown_connections 2024-09-16T01:26:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 1 -- 172.21.15.78:0/1448217771 wait complete. 2024-09-16T01:26:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073dd5c740 20 client.17211 trim_cache size 0 max 16384 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 31, 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "leader": 24485, 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "set_version": 60, 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "sets": {} 2024-09-16T01:26:39.465 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 msgr2=0x7f071c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 0x7f071c0a5870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f072c005ab0 tx=0x7f072c009260 comp rx=0 tx=0).stop 2024-09-16T01:26:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 msgr2=0x7f07341983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f072400ba50 tx=0x7f072400bf10 comp rx=0 tx=0).stop 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 shutdown_connections 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f071c0a33b0 0x7f071c0a5870 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0734106a80 0x7f073419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0734101420 0x7f0734198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 --2- 172.21.15.78:0/3772168089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0734100a80 0x7f07341983d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.440+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 >> 172.21.15.78:0/3772168089 conn(0x7f07340fc7a0 msgr2=0x7f0734104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.441+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 shutdown_connections 2024-09-16T01:26:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:39.441+0000 7f073b983640 1 -- 172.21.15.78:0/3772168089 wait complete. 2024-09-16T01:26:39.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:39 smithi193 ceph-mon[35701]: pgmap v499: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-09-16T01:26:39.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:39 smithi148 ceph-mon[35030]: pgmap v499: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.0 MiB/s wr, 650 op/s 2024-09-16T01:26:40.202 INFO:tasks.quiescer:done joining Quiescers 2024-09-16T01:26:40.203 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-16T01:26:40.212 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:26:40.221 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-16T01:26:40.232 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-16T01:26:40.232 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T01:26:40.232 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:26:40.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-16T01:26:40.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-16T01:26:40.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-16T01:26:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 -- 172.21.15.78:0/3245416685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc101370 msgr2=0x7fafbc10d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 --2- 172.21.15.78:0/3245416685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc101370 0x7fafbc10d8a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009cd0 tx=0x7fafb402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 -- 172.21.15.78:0/3245416685 shutdown_connections 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 --2- 172.21.15.78:0/3245416685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc101370 0x7fafbc10d8a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 --2- 172.21.15.78:0/3245416685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 0x7fafbc100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 --2- 172.21.15.78:0/3245416685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc1069e0 0x7fafbc106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 -- 172.21.15.78:0/3245416685 >> 172.21.15.78:0/3245416685 conn(0x7fafbc0fc6f0 msgr2=0x7fafbc0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 -- 172.21.15.78:0/3245416685 shutdown_connections 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.476+0000 7fafc45f6640 1 -- 172.21.15.78:0/3245416685 wait complete. 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 Processor -- start 2024-09-16T01:26:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 -- start start 2024-09-16T01:26:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 0x7fafbc196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc101370 0x7fafbc197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc110e20 con 0x7fafbc1069e0 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc110ca0 con 0x7fafbc101370 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc45f6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc110fa0 con 0x7fafbc1009d0 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39796/0 (socket says 172.21.15.78:39796) 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 -- 172.21.15.78:0/1899286154 learned_addr learned my addr 172.21.15.78:0/1899286154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc1b91640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc101370 0x7fafbc197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2392640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 0x7fafbc196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 -- 172.21.15.78:0/1899286154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 msgr2=0x7fafbc196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 0x7fafbc196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 -- 172.21.15.78:0/1899286154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc101370 msgr2=0x7fafbc197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc101370 0x7fafbc197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.477+0000 7fafc2b93640 1 -- 172.21.15.78:0/1899286154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafbc1a66b0 con 0x7fafbc1069e0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafc2b93640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006010 tx=0x7fafb4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4004450 con 0x7fafbc1069e0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb4003e10 con 0x7fafbc1069e0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafbc1a6940 con 0x7fafbc1069e0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb40466e0 con 0x7fafbc1069e0 2024-09-16T01:26:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.478+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafbc1a6e90 con 0x7fafbc1069e0 2024-09-16T01:26:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.479+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafbc102110 con 0x7fafbc1069e0 2024-09-16T01:26:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.480+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fafb40305a0 con 0x7fafbc1069e0 2024-09-16T01:26:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.481+0000 7fafb37fe640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 0x7fafa80a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.481+0000 7fafc2392640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 0x7fafa80a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.481+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7fafb402e030 con 0x7fafbc1069e0 2024-09-16T01:26:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.481+0000 7fafc2392640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 0x7fafa80a5920 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafac00cac0 tx=0x7fafac00c4a0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.484+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fafb404b060 con 0x7fafbc1069e0 2024-09-16T01:26:40.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.673+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fafbc0fd1e0 con 0x7fafa80a3460 2024-09-16T01:26:40.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.680+0000 7fafb37fe640 1 -- 172.21.15.78:0/1899286154 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7fafbc0fd1e0 con 0x7fafa80a3460 2024-09-16T01:26:40.681 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-16T00:00:00", "created": "2024-09-16T01:10:49", "first": "2024-09-16T01:11:00", "last": "2024-09-16T01:26:00", "last_pruned": "2024-09-16T01:26:00", "created_count": 16, "pruned_count": 10, "active": true} 2024-09-16T01:26:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 msgr2=0x7fafa80a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 0x7fafa80a5920 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafac00cac0 tx=0x7fafac00c4a0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 msgr2=0x7fafbc1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fafb4006010 tx=0x7fafb4003c20 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 shutdown_connections 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fafa80a3460 0x7fafa80a5920 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafbc1069e0 0x7fafbc1978c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafbc101370 0x7fafbc197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 --2- 172.21.15.78:0/1899286154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc1009d0 0x7fafbc196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 >> 172.21.15.78:0/1899286154 conn(0x7fafbc0fc6f0 msgr2=0x7fafbc0fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 shutdown_connections 2024-09-16T01:26:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.682+0000 7fafc45f6640 1 -- 172.21.15.78:0/1899286154 wait complete. 2024-09-16T01:26:40.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-16T01:26:40.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- 172.21.15.78:0/1217042141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe8101370 msgr2=0x7f7fe810d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 --2- 172.21.15.78:0/1217042141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe8101370 0x7f7fe810d8a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009cd0 tx=0x7f7fe002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- 172.21.15.78:0/1217042141 shutdown_connections 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 --2- 172.21.15.78:0/1217042141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe8101370 0x7f7fe810d8a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 --2- 172.21.15.78:0/1217042141 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe81009d0 0x7f7fe8100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 --2- 172.21.15.78:0/1217042141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81069e0 0x7f7fe8106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- 172.21.15.78:0/1217042141 >> 172.21.15.78:0/1217042141 conn(0x7f7fe80fc6f0 msgr2=0x7f7fe80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- 172.21.15.78:0/1217042141 shutdown_connections 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- 172.21.15.78:0/1217042141 wait complete. 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 Processor -- start 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.794+0000 7f7feeed7640 1 -- start start 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81009d0 0x7f7fe8197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 0x7f7fe8197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8110e20 con 0x7f7fe81069e0 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8110ca0 con 0x7f7fe8101370 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8110fa0 con 0x7f7fe81009d0 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39818/0 (socket says 172.21.15.78:39818) 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 -- 172.21.15.78:0/3739108819 learned_addr learned my addr 172.21.15.78:0/3739108819 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fe7fff640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 0x7f7fe8197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fecc73640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81009d0 0x7f7fe8197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 -- 172.21.15.78:0/3739108819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81009d0 msgr2=0x7f7fe8197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81009d0 0x7f7fe8197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 -- 172.21.15.78:0/3739108819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 msgr2=0x7f7fe8197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 0x7f7fe8197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 -- 172.21.15.78:0/3739108819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe819c140 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fe7fff640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 0x7f7fe8197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fed474640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0006010 tx=0x7f7fe0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0004450 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe0003e10 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe819c3d0 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.795+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe00466e0 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.796+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe81a3cc0 con 0x7f7fe81069e0 2024-09-16T01:26:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.796+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe8100e30 con 0x7f7fe81069e0 2024-09-16T01:26:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.798+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7fe00305a0 con 0x7f7fe81069e0 2024-09-16T01:26:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.799+0000 7f7fe5ffb640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 0x7f7fd40a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.799+0000 7f7fecc73640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 0x7f7fd40a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:40.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.799+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f7fe002e030 con 0x7f7fe81069e0 2024-09-16T01:26:40.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.799+0000 7f7fecc73640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 0x7f7fd40a5920 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000b9c0 tx=0x7f7fd0005c50 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.802+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7fe004b060 con 0x7f7fe81069e0 2024-09-16T01:26:40.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.989+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 --> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f7fe80fe300 con 0x7f7fd40a3460 2024-09-16T01:26:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.996+0000 7f7fe5ffb640 1 -- 172.21.15.78:0/3739108819 <== mgr.15225 v2:172.21.15.78:6800/2602873484 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f7fe80fe300 con 0x7f7fd40a3460 2024-09-16T01:26:40.997 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1m 6m3h 2024-09-16T01:26:40.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 msgr2=0x7f7fd40a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 0x7f7fd40a5920 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000b9c0 tx=0x7f7fd0005c50 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 msgr2=0x7f7fe819ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0006010 tx=0x7f7fe0003c20 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 shutdown_connections 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7fd40a3460 0x7f7fd40a5920 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fe81069e0 0x7f7fe819ba10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fe8101370 0x7f7fe8197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 --2- 172.21.15.78:0/3739108819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7fe81009d0 0x7f7fe8197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 >> 172.21.15.78:0/3739108819 conn(0x7f7fe80fc6f0 msgr2=0x7f7fe80fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 shutdown_connections 2024-09-16T01:26:41.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:40.998+0000 7f7feeed7640 1 -- 172.21.15.78:0/3739108819 wait complete. 2024-09-16T01:26:41.010 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-16T01:26:41.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:26:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 -- 172.21.15.78:0/1167340612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c78103610 msgr2=0x7f1c7810baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/1167340612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c78103610 0x7f1c7810baf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009900 tx=0x7f1c6802a480 comp rx=0 tx=0).stop 2024-09-16T01:26:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 -- 172.21.15.78:0/1167340612 shutdown_connections 2024-09-16T01:26:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/1167340612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c7810c0c0 0x7f1c7810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/1167340612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c78103610 0x7f1c7810baf0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/1167340612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 0x7f1c781030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 -- 172.21.15.78:0/1167340612 >> 172.21.15.78:0/1167340612 conn(0x7f1c78077a10 msgr2=0x7f1c780fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 -- 172.21.15.78:0/1167340612 shutdown_connections 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 -- 172.21.15.78:0/1167340612 wait complete. 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.127+0000 7f1c7f41e640 1 Processor -- start 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 -- start start 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 0x7f1c781983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c78103610 0x7f1c78198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78111e90 con 0x7f1c7810c0c0 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78111d10 con 0x7f1c78103610 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7f41e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1c78112010 con 0x7f1c78102cf0 2024-09-16T01:26:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39842/0 (socket says 172.21.15.78:39842) 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 -- 172.21.15.78:0/4240832036 learned_addr learned my addr 172.21.15.78:0/4240832036 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d1ba640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 0x7f1c781983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7c9b9640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c78103610 0x7f1c78198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 -- 172.21.15.78:0/4240832036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 msgr2=0x7f1c781983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 0x7f1c781983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 -- 172.21.15.78:0/4240832036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c78103610 msgr2=0x7f1c78198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c78103610 0x7f1c78198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 -- 172.21.15.78:0/4240832036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7819d3d0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.128+0000 7f1c7d9bb640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009fd0 tx=0x7f1c7000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70017070 con 0x7f1c7810c0c0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7819d6c0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c70003e50 con 0x7f1c7810c0c0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c700106c0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c781a4f50 con 0x7f1c7810c0c0 2024-09-16T01:26:41.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.129+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c781030d0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.131+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f1c70003970 con 0x7f1c7810c0c0 2024-09-16T01:26:41.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.131+0000 7f1c6e7fc640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 0x7f1c640a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.132+0000 7f1c7d1ba640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 0x7f1c640a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.132+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f1c700cfdd0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.132+0000 7f1c7d1ba640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 0x7f1c640a5880 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400bbe0 tx=0x7f1c7400b390 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.139+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1c7009c3c0 con 0x7f1c7810c0c0 2024-09-16T01:26:41.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:41 smithi078 ceph-mon[30937]: pgmap v500: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-09-16T01:26:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.315+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c78199730 con 0x7f1c7810c0c0 2024-09-16T01:26:41.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.317+0000 7f1c6e7fc640 1 -- 172.21.15.78:0/4240832036 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+25409 (secure 0 0 0) 0x7f1c78199730 con 0x7f1c7810c0c0 2024-09-16T01:26:41.318 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:41.319 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":115,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:26:20.622226+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1816200719":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:0/557237582":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1671709190":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3382939699":"2024-09-17T01:10:47.191866+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]},{"pool":4,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:26:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 msgr2=0x7f1c640a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 0x7f1c640a5880 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400bbe0 tx=0x7f1c7400b390 comp rx=0 tx=0).stop 2024-09-16T01:26:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 msgr2=0x7f1c7819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009fd0 tx=0x7f1c7000b260 comp rx=0 tx=0).stop 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 shutdown_connections 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1c640a33c0 0x7f1c640a5880 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c7810c0c0 0x7f1c7819cca0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c78103610 0x7f1c78198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 --2- 172.21.15.78:0/4240832036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1c78102cf0 0x7f1c781983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 >> 172.21.15.78:0/4240832036 conn(0x7f1c78077a10 msgr2=0x7f1c780fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 shutdown_connections 2024-09-16T01:26:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.319+0000 7f1c7f41e640 1 -- 172.21.15.78:0/4240832036 wait complete. 2024-09-16T01:26:41.331 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:03:52.230028+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:08:25.352496+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:08:27.537592+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-16T01:08:33.982278+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-16T01:26:41.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:26:41.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- 172.21.15.78:0/1200646029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c100980 msgr2=0x7f1b6c100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/1200646029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c100980 0x7f1b6c100de0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009930 tx=0x7f1b6802a420 comp rx=0 tx=0).stop 2024-09-16T01:26:41.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- 172.21.15.78:0/1200646029 shutdown_connections 2024-09-16T01:26:41.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/1200646029 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c101320 0x7f1b6c10d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/1200646029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c100980 0x7f1b6c100de0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/1200646029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c106990 0x7f1b6c106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- 172.21.15.78:0/1200646029 >> 172.21.15.78:0/1200646029 conn(0x7f1b6c0779d0 msgr2=0x7f1b6c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- 172.21.15.78:0/1200646029 shutdown_connections 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- 172.21.15.78:0/1200646029 wait complete. 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 Processor -- start 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- start start 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c100980 0x7f1b6c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c106990 0x7f1b6c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c110e20 con 0x7f1b6c101320 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c110ca0 con 0x7f1b6c106990 2024-09-16T01:26:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.436+0000 7f1b73bf6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c110fa0 con 0x7f1b6c100980 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39860/0 (socket says 172.21.15.78:39860) 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 -- 172.21.15.78:0/2830203587 learned_addr learned my addr 172.21.15.78:0/2830203587 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71992640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c100980 0x7f1b6c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b72193640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c106990 0x7f1b6c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 -- 172.21.15.78:0/2830203587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c100980 msgr2=0x7f1b6c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c100980 0x7f1b6c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 -- 172.21.15.78:0/2830203587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c106990 msgr2=0x7f1b6c19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c106990 0x7f1b6c19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 -- 172.21.15.78:0/2830203587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c19d3a0 con 0x7f1b6c101320 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b71191640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1b6802a940 tx=0x7f1b68003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6803f070 con 0x7f1b6c101320 2024-09-16T01:26:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b68031c90 con 0x7f1b6c101320 2024-09-16T01:26:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6c19d630 con 0x7f1b6c101320 2024-09-16T01:26:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68035030 con 0x7f1b6c101320 2024-09-16T01:26:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.437+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6c1a4f20 con 0x7f1b6c101320 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.438+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6c1020c0 con 0x7f1b6c101320 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.440+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f1b68042070 con 0x7f1b6c101320 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.441+0000 7f1b62ffd640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 0x7f1b580a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.441+0000 7f1b71992640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 0x7f1b580a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.441+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f1b680f0bf0 con 0x7f1b6c101320 2024-09-16T01:26:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.441+0000 7f1b71992640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 0x7f1b580a5880 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00bb60 tx=0x7f1b5c00b390 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.444+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b680bcca0 con 0x7f1b6c101320 2024-09-16T01:26:41.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:41 smithi193 ceph-mon[35701]: pgmap v500: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-09-16T01:26:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.618+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b6c1a5120 con 0x7f1b6c101320 2024-09-16T01:26:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.619+0000 7f1b62ffd640 1 -- 172.21.15.78:0/2830203587 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7f1b680c0fa0 con 0x7f1b6c101320 2024-09-16T01:26:41.620 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-09-16T01:26:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 msgr2=0x7f1b580a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 0x7f1b580a5880 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00bb60 tx=0x7f1b5c00b390 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 msgr2=0x7f1b6c1988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1b6802a940 tx=0x7f1b68003b90 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 shutdown_connections 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f1b580a33c0 0x7f1b580a5880 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b6c106990 0x7f1b6c19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c101320 0x7f1b6c1988f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 --2- 172.21.15.78:0/2830203587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6c100980 0x7f1b6c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 >> 172.21.15.78:0/2830203587 conn(0x7f1b6c0779d0 msgr2=0x7f1b6c0694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 shutdown_connections 2024-09-16T01:26:41.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.621+0000 7f1b73bf6640 1 -- 172.21.15.78:0/2830203587 wait complete. 2024-09-16T01:26:41.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-16T01:26:41.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:41 smithi148 ceph-mon[35030]: pgmap v500: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- 172.21.15.78:0/1908599175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 msgr2=0x7f977c106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 --2- 172.21.15.78:0/1908599175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c106e10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9778009890 tx=0x7f977802a380 comp rx=0 tx=0).stop 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- 172.21.15.78:0/1908599175 shutdown_connections 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 --2- 172.21.15.78:0/1908599175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 0x7f977c10d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 --2- 172.21.15.78:0/1908599175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 --2- 172.21.15.78:0/1908599175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c106e10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- 172.21.15.78:0/1908599175 >> 172.21.15.78:0/1908599175 conn(0x7f977c0fc740 msgr2=0x7f977c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- 172.21.15.78:0/1908599175 shutdown_connections 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- 172.21.15.78:0/1908599175 wait complete. 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 Processor -- start 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.740+0000 7f9783865640 1 -- start start 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 0x7f977c1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f977c110e30 con 0x7f977c106a30 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f977c110cb0 con 0x7f977c1013c0 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9783865640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f977c110fb0 con 0x7f977c100a20 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781e02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781e02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39878/0 (socket says 172.21.15.78:39878) 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781e02640 1 -- 172.21.15.78:0/119913260 learned_addr learned my addr 172.21.15.78:0/119913260 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9780e00640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 0x7f977c1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 -- 172.21.15.78:0/119913260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 msgr2=0x7f977c1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 0x7f977c1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 -- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 msgr2=0x7f977c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 -- 172.21.15.78:0/119913260 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c1a47b0 con 0x7f977c100a20 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781e02640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.741+0000 7f9781601640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c1993b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f977802a8a0 tx=0x7f9778033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.742+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977803f020 con 0x7f977c100a20 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.742+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977c1a4a40 con 0x7f977c100a20 2024-09-16T01:26:41.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.742+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9778004240 con 0x7f977c100a20 2024-09-16T01:26:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.742+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778037040 con 0x7f977c100a20 2024-09-16T01:26:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.742+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977c1a4f80 con 0x7f977c100a20 2024-09-16T01:26:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.743+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977c100e80 con 0x7f977c100a20 2024-09-16T01:26:41.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.744+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f9778042020 con 0x7f977c100a20 2024-09-16T01:26:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.745+0000 7f97727fc640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 0x7f97680a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.745+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f97780f0490 con 0x7f977c100a20 2024-09-16T01:26:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.745+0000 7f9780e00640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 0x7f97680a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:41.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.745+0000 7f9780e00640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 0x7f97680a5880 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f977c0ffd00 tx=0x7f976c009260 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:41.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.748+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9778049020 con 0x7f977c100a20 2024-09-16T01:26:41.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.922+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f977c108ff0 con 0x7f977c100a20 2024-09-16T01:26:41.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.923+0000 7f97727fc640 1 -- 172.21.15.78:0/119913260 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) ==== 104+0+11 (secure 0 0 0) 0x7f97780bcb90 con 0x7f977c100a20 2024-09-16T01:26:41.924 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 msgr2=0x7f97680a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 0x7f97680a5880 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f977c0ffd00 tx=0x7f976c009260 comp rx=0 tx=0).stop 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 msgr2=0x7f977c1993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c1993b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f977802a8a0 tx=0x7f9778033d90 comp rx=0 tx=0).stop 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 shutdown_connections 2024-09-16T01:26:41.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f97680a33c0 0x7f97680a5880 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f977c106a30 0x7f977c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c1013c0 0x7f977c1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 --2- 172.21.15.78:0/119913260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100a20 0x7f977c1993b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 >> 172.21.15.78:0/119913260 conn(0x7f977c0fc740 msgr2=0x7f977c0fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 shutdown_connections 2024-09-16T01:26:41.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:41.925+0000 7f9783865640 1 -- 172.21.15.78:0/119913260 wait complete. 2024-09-16T01:26:41.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-16T01:26:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 -- 172.21.15.78:0/3592578927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c102be0 msgr2=0x7f786c102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 --2- 172.21.15.78:0/3592578927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c102be0 0x7f786c102fc0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f785c009900 tx=0x7f785c02a480 comp rx=0 tx=0).stop 2024-09-16T01:26:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 -- 172.21.15.78:0/3592578927 shutdown_connections 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 --2- 172.21.15.78:0/3592578927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c10bfb0 0x7f786c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 --2- 172.21.15.78:0/3592578927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c103500 0x7f786c10b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.040+0000 7f786b59e640 1 --2- 172.21.15.78:0/3592578927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c102be0 0x7f786c102fc0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 -- 172.21.15.78:0/3592578927 >> 172.21.15.78:0/3592578927 conn(0x7f786c0778c0 msgr2=0x7f786c0fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 -- 172.21.15.78:0/3592578927 shutdown_connections 2024-09-16T01:26:42.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 -- 172.21.15.78:0/3592578927 wait complete. 2024-09-16T01:26:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 Processor -- start 2024-09-16T01:26:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 -- start start 2024-09-16T01:26:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c102be0 0x7f786c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.041+0000 7f786b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c103500 0x7f786c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f786c111d80 con 0x7f786c10bfb0 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f786c111c00 con 0x7f786c103500 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f786c111f00 con 0x7f786c102be0 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39896/0 (socket says 172.21.15.78:39896) 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 -- 172.21.15.78:0/2963754617 learned_addr learned my addr 172.21.15.78:0/2963754617 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786a59c640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c102be0 0x7f786c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f7869d9b640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c103500 0x7f786c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 -- 172.21.15.78:0/2963754617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c102be0 msgr2=0x7f786c1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c102be0 0x7f786c1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 -- 172.21.15.78:0/2963754617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c103500 msgr2=0x7f786c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c103500 0x7f786c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 -- 172.21.15.78:0/2963754617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c19d2c0 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786ad9d640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f786400eb10 tx=0x7f786400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864004b20 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c19d5b0 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7864004cc0 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864010760 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.042+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1a4e40 con 0x7f786c10bfb0 2024-09-16T01:26:42.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.043+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c102fc0 con 0x7f786c10bfb0 2024-09-16T01:26:42.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.045+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7864010900 con 0x7f786c10bfb0 2024-09-16T01:26:42.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.045+0000 7f785b7fe640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 0x7f78540a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.045+0000 7f786a59c640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 0x7f78540a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.045+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f78640d0830 con 0x7f786c10bfb0 2024-09-16T01:26:42.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.046+0000 7f786a59c640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 0x7f78540a5880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f785c009230 tx=0x7f785c0058d0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.052+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f78640541b0 con 0x7f786c10bfb0 2024-09-16T01:26:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.229+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f786c107060 con 0x7f786c10bfb0 2024-09-16T01:26:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.229+0000 7f785b7fe640 1 -- 172.21.15.78:0/2963754617 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) ==== 100+0+11 (secure 0 0 0) 0x7f786409ce20 con 0x7f786c10bfb0 2024-09-16T01:26:42.230 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 msgr2=0x7f78540a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 0x7f78540a5880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f785c009230 tx=0x7f785c0058d0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 msgr2=0x7f786c19cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f786400eb10 tx=0x7f786400c2d0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 shutdown_connections 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f78540a33c0 0x7f78540a5880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786c10bfb0 0x7f786c19cb90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f786c103500 0x7f786c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 --2- 172.21.15.78:0/2963754617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f786c102be0 0x7f786c1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 >> 172.21.15.78:0/2963754617 conn(0x7f786c0778c0 msgr2=0x7f786c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 shutdown_connections 2024-09-16T01:26:42.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.231+0000 7f786b59e640 1 -- 172.21.15.78:0/2963754617 wait complete. 2024-09-16T01:26:42.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-16T01:26:42.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 -- 172.21.15.78:0/3859548730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f750410e420 msgr2=0x7f7504110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 --2- 172.21.15.78:0/3859548730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f750410e420 0x7f7504110810 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009be0 tx=0x7f74fc02a6c0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 -- 172.21.15.78:0/3859548730 shutdown_connections 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 --2- 172.21.15.78:0/3859548730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f750410e420 0x7f7504110810 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 --2- 172.21.15.78:0/3859548730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7504101190 0x7f750410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 --2- 172.21.15.78:0/3859548730 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7504100870 0x7f7504100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 -- 172.21.15.78:0/3859548730 >> 172.21.15.78:0/3859548730 conn(0x7f75040779d0 msgr2=0x7f75040fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 -- 172.21.15.78:0/3859548730 shutdown_connections 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.347+0000 7f750b067640 1 -- 172.21.15.78:0/3859548730 wait complete. 2024-09-16T01:26:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 Processor -- start 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 -- start start 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7504100870 0x7f750419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 0x7f750419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7504114050 con 0x7f7504101190 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7504113ed0 con 0x7f750410e420 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f750b067640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75041141d0 con 0x7f7504100870 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f7503fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f7503fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39908/0 (socket says 172.21.15.78:39908) 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f7503fff640 1 -- 172.21.15.78:0/317549825 learned_addr learned my addr 172.21.15.78:0/317549825 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.348+0000 7f7509604640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 0x7f750419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 -- 172.21.15.78:0/317549825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7504100870 msgr2=0x7f750419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7504100870 0x7f750419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 -- 172.21.15.78:0/317549825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 msgr2=0x7f750419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 0x7f750419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 -- 172.21.15.78:0/317549825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419f5b0 con 0x7f7504101190 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7509604640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 0x7f750419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7503fff640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9c0 tx=0x7f74f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f000ccc0 con 0x7f7504101190 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0005500 con 0x7f7504101190 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0010430 con 0x7f7504101190 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419f8a0 con 0x7f7504101190 2024-09-16T01:26:42.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.349+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a7130 con 0x7f7504101190 2024-09-16T01:26:42.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.350+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7504100c50 con 0x7f7504101190 2024-09-16T01:26:42.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.351+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f74f00046c0 con 0x7f7504101190 2024-09-16T01:26:42.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.352+0000 7f7501ffb640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 0x7f74f40a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.352+0000 7f7508e03640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 0x7f74f40a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.352+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f74f00d04b0 con 0x7f7504101190 2024-09-16T01:26:42.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.353+0000 7f7508e03640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 0x7f74f40a5880 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74f80058c0 tx=0x7f74f8005850 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.356+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f74f009caa0 con 0x7f7504101190 2024-09-16T01:26:42.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:42 smithi078 ceph-mon[30937]: from='client.17217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:42 smithi078 ceph-mon[30937]: from='client.17223 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4240832036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:26:42.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2830203587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:42 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/119913260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.528+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7504109610 con 0x7f7504101190 2024-09-16T01:26:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.528+0000 7f7501ffb640 1 -- 172.21.15.78:0/317549825 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v115) ==== 103+0+11 (secure 0 0 0) 0x7f74f009c3a0 con 0x7f7504101190 2024-09-16T01:26:42.530 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 msgr2=0x7f74f40a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 0x7f74f40a5880 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74f80058c0 tx=0x7f74f8005850 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 msgr2=0x7f750419ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9c0 tx=0x7f74f000ee80 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 shutdown_connections 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f74f40a33c0 0x7f74f40a5880 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f750410e420 0x7f750419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7504101190 0x7f750419ab00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 --2- 172.21.15.78:0/317549825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7504100870 0x7f750419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 >> 172.21.15.78:0/317549825 conn(0x7f75040779d0 msgr2=0x7f75040fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.530+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 shutdown_connections 2024-09-16T01:26:42.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.531+0000 7f750b067640 1 -- 172.21.15.78:0/317549825 wait complete. 2024-09-16T01:26:42.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-16T01:26:42.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:42 smithi193 ceph-mon[35701]: from='client.17217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:42 smithi193 ceph-mon[35701]: from='client.17223 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4240832036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:26:42.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2830203587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:42 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/119913260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:42 smithi148 ceph-mon[35030]: from='client.17217 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:42 smithi148 ceph-mon[35030]: from='client.17223 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-16T01:26:42.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4240832036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:26:42.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2830203587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:42 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/119913260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 -- 172.21.15.78:0/2826069044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 msgr2=0x7f00d4103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 --2- 172.21.15.78:0/2826069044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4103050 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f00c4009900 tx=0x7f00c402a480 comp rx=0 tx=0).stop 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 -- 172.21.15.78:0/2826069044 shutdown_connections 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 --2- 172.21.15.78:0/2826069044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00d410c090 0x7f00d410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 --2- 172.21.15.78:0/2826069044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d4103590 0x7f00d410ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 --2- 172.21.15.78:0/2826069044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4103050 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 -- 172.21.15.78:0/2826069044 >> 172.21.15.78:0/2826069044 conn(0x7f00d40fc6c0 msgr2=0x7f00d40feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 -- 172.21.15.78:0/2826069044 shutdown_connections 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.641+0000 7f00da6a6640 1 -- 172.21.15.78:0/2826069044 wait complete. 2024-09-16T01:26:42.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 Processor -- start 2024-09-16T01:26:42.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 -- start start 2024-09-16T01:26:42.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00d4103590 0x7f00d4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d410c090 0x7f00d419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4111eb0 con 0x7f00d4102c70 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4111d30 con 0x7f00d410c090 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00da6a6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4112030 con 0x7f00d4103590 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39916/0 (socket says 172.21.15.78:39916) 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.642+0000 7f00d3fff640 1 -- 172.21.15.78:0/3830368458 learned_addr learned my addr 172.21.15.78:0/3830368458 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d8c43640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d410c090 0x7f00d419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 -- 172.21.15.78:0/3830368458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00d4103590 msgr2=0x7f00d4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00d4103590 0x7f00d4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 -- 172.21.15.78:0/3830368458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d410c090 msgr2=0x7f00d419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d410c090 0x7f00d419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:42.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 -- 172.21.15.78:0/3830368458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d419d410 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d3fff640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f00c4006010 tx=0x7f00c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c403f030 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00c4003ce0 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d419d6a0 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4035070 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.643+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d41a4f90 con 0x7f00d4102c70 2024-09-16T01:26:42.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.644+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d4103050 con 0x7f00d4102c70 2024-09-16T01:26:42.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.645+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f00c4042030 con 0x7f00d4102c70 2024-09-16T01:26:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.646+0000 7f00d17fa640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 0x7f00bc0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:26:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.646+0000 7f00d37fe640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 0x7f00bc0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:26:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.646+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9525+0+0 (secure 0 0 0) 0x7f00c40f0930 con 0x7f00d4102c70 2024-09-16T01:26:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.646+0000 7f00d37fe640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 0x7f00bc0a5880 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f00c8005fd0 tx=0x7f00c8005c70 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:26:42.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.649+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f00c40bcfe0 con 0x7f00d4102c70 2024-09-16T01:26:42.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.864+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f00d41070f0 con 0x7f00d4102c70 2024-09-16T01:26:42.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.871+0000 7f00d17fa640 1 -- 172.21.15.78:0/3830368458 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+742872 (secure 0 0 0) 0x7f00c40bc8e0 con 0x7f00d4102c70 2024-09-16T01:26:42.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:26:42.884 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:42.884 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 26, 2024-09-16T01:26:42.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "active_gid": 15225, 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "active_name": "x", 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addrs": { 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2602873484 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2602873484 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addr": "172.21.15.78:6801/2602873484", 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "active_change": "2024-09-16T01:10:47.192093+0000", 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "active_mgr_features": 4540701547738038271, 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "available": true, 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: "standbys": [ 2024-09-16T01:26:42.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 24764, 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "y", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.889 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:42.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.895 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-09-16T01:26:42.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-09-16T01:26:42.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.902 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-09-16T01:26:42.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.906 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:42.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-09-16T01:26:42.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-09-16T01:26:42.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.916 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:42.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:42.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.920 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-09-16T01:26:42.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-09-16T01:26:42.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T01:26:42.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.929 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:42.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T01:26:42.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-09-16T01:26:42.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-09-16T01:26:42.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.939 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.941 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-09-16T01:26:42.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-09-16T01:26:42.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T01:26:42.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.952 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-09-16T01:26:42.955 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-09-16T01:26:42.956 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-09-16T01:26:42.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.959 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_samba": { 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_samba", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T01:26:42.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Samba/SMB container image", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-09-16T01:26:42.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T01:26:42.964 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-09-16T01:26:42.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T01:26:42.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-09-16T01:26:42.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-09-16T01:26:42.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "grafana_dashboards_path": { 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "grafana_dashboards_path", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T01:26:42.975 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:42.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-09-16T01:26:42.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.978 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.980 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T01:26:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-09-16T01:26:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-09-16T01:26:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-09-16T01:26:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-09-16T01:26:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T01:26:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.013 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-09-16T01:26:43.015 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.016 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.017 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.020 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-09-16T01:26:43.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.026 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.030 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.033 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T01:26:43.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T01:26:43.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.036 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.039 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.042 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T01:26:43.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T01:26:43.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.047 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.053 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.055 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T01:26:43.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T01:26:43.060 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.062 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.065 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.067 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.069 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T01:26:43.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.072 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.075 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.076 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.080 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.083 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.086 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-09-16T01:26:43.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.098 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.099 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.100 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.110 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.112 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.113 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.114 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.115 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.116 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.121 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.124 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.128 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.129 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.130 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.132 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.136 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.138 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.140 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.141 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.142 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.144 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.146 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.147 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.152 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.154 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.155 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.156 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.160 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.161 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.162 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.163 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-09-16T01:26:43.166 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T01:26:43.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.171 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.173 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.175 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.177 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.182 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.183 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.184 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.186 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.190 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.192 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.193 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-09-16T01:26:43.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-09-16T01:26:43.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.197 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.198 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.199 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.204 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.206 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.208 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.209 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.210 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.212 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.215 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.218 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.222 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-09-16T01:26:43.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.225 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-09-16T01:26:43.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.230 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.237 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-09-16T01:26:43.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.240 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:43.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.244 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.245 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.247 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.252 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-09-16T01:26:43.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:43.254 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.256 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.259 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.261 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-09-16T01:26:43.263 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.264 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.266 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.267 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.269 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.271 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.272 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.274 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.278 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.280 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.281 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.282 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-09-16T01:26:43.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-09-16T01:26:43.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.292 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.295 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.296 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.297 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.298 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.299 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.305 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.306 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.307 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.308 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.309 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.310 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.311 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.313 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-09-16T01:26:43.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.319 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.320 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.322 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.324 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.326 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-09-16T01:26:43.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-09-16T01:26:43.328 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.329 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.330 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.332 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.333 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-09-16T01:26:43.334 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-09-16T01:26:43.335 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.340 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.341 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.342 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.345 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.346 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.348 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.352 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.356 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.359 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.360 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-09-16T01:26:43.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.363 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T01:26:43.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.366 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.367 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.368 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.371 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.372 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.375 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.377 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 24853, 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540701547738038271, 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.379 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.381 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.382 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.384 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.385 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.387 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.390 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-09-16T01:26:43.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-09-16T01:26:43.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.393 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.394 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.395 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-09-16T01:26:43.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-09-16T01:26:43.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-09-16T01:26:43.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-09-16T01:26:43.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.404 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.408 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.409 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-09-16T01:26:43.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T01:26:43.413 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.415 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-09-16T01:26:43.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.417 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.418 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T01:26:43.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.420 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.421 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.424 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.426 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T01:26:43.427 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.428 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.429 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.430 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.431 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-09-16T01:26:43.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-09-16T01:26:43.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.436 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.440 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.441 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_samba": { 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_samba", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.443 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Samba/SMB container image", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T01:26:43.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.446 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T01:26:43.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.450 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "grafana_dashboards_path": { 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "grafana_dashboards_path", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.453 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.454 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.455 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.456 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.457 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T01:26:43.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.461 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T01:26:43.463 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.464 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.466 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-09-16T01:26:43.469 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.470 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-09-16T01:26:43.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.473 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.475 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T01:26:43.477 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-09-16T01:26:43.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T01:26:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T01:26:43.480 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T01:26:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T01:26:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T01:26:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T01:26:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T01:26:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T01:26:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-09-16T01:26:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-09-16T01:26:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-09-16T01:26:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-09-16T01:26:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-09-16T01:26:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-09-16T01:26:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-09-16T01:26:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T01:26:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-09-16T01:26:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T01:26:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-09-16T01:26:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-09-16T01:26:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-09-16T01:26:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T01:26:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-09-16T01:26:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-09-16T01:26:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-09-16T01:26:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.755 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-09-16T01:26:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-09-16T01:26:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-09-16T01:26:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-09-16T01:26:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-09-16T01:26:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-09-16T01:26:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-09-16T01:26:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "modules": [ 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard", 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "iostat", 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "nfs", 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "restful", 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_schedule" 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-09-16T01:26:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-09-16T01:26:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-09-16T01:26:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T01:26:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T01:26:43.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-09-16T01:26:43.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-09-16T01:26:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "read", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap-read" 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-09-16T01:26:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-09-16T01:26:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_log_destination": { 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm_log_destination", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "file", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "file,syslog", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "syslog" 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-09-16T01:26:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-09-16T01:26:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-16T01:26:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-09-16T01:26:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-09-16T01:26:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-09-16T01:26:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_samba": { 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_samba", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Samba/SMB container image", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-09-16T01:26:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-09-16T01:26:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "grafana_dashboards_path": { 2024-09-16T01:26:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "grafana_dashboards_path", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-09-16T01:26:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T01:26:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-09-16T01:26:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T01:26:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T01:26:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T01:26:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T01:26:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-09-16T01:26:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T01:26:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T01:26:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-09-16T01:26:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-09-16T01:26:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-09-16T01:26:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-09-16T01:26:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-09-16T01:26:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-09-16T01:26:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-09-16T01:26:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-09-16T01:26:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-09-16T01:26:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-09-16T01:26:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-09-16T01:26:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-09-16T01:26:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-09-16T01:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-09-16T01:26:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-09-16T01:26:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-09-16T01:26:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-09-16T01:26:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.156 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-09-16T01:26:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-09-16T01:26:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-09-16T01:26:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-09-16T01:26:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-09-16T01:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-09-16T01:26:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-09-16T01:26:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-09-16T01:26:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T01:26:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-09-16T01:26:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-09-16T01:26:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-09-16T01:26:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "periodic_async_work": { 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "periodic_async_work", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Periodically check for async work", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_no_wait": { 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T01:26:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-09-16T01:26:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-09-16T01:26:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-09-16T01:26:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T01:26:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "services": { 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard": "https://172.21.15.78:8443/" 2024-09-16T01:26:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "always_on_modules": { 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus": [ 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific": [ 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy": [ 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-09-16T01:26:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "reef": [ 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-09-16T01:26:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "squid": [ 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-09-16T01:26:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "last_failure_osd_epoch": 94, 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "active_clients": [ 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2104614763 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "libcephsqlite", 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2535367500 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1475124172 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4007821665 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2413750399 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 927937356 2024-09-16T01:26:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-16T01:26:44.318 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:44.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:43 smithi148 ceph-mon[35030]: pgmap v501: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 4.8 MiB/s wr, 519 op/s 2024-09-16T01:26:44.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2963754617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/317549825' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.319 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:43 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/3830368458' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T01:26:44.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:43 smithi193 ceph-mon[35701]: pgmap v501: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 4.8 MiB/s wr, 519 op/s 2024-09-16T01:26:44.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2963754617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/317549825' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:43 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/3830368458' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T01:26:44.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.876+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 msgr2=0x7f00bc0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.876+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 0x7f00bc0a5880 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f00c8005fd0 tx=0x7f00c8005c70 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.876+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 msgr2=0x7f00d4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f00c4006010 tx=0x7f00c40042f0 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 shutdown_connections 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f00bc0a33c0 0x7f00bc0a5880 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00d410c090 0x7f00d419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00d4103590 0x7f00d4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 --2- 172.21.15.78:0/3830368458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4102c70 0x7f00d4198420 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 >> 172.21.15.78:0/3830368458 conn(0x7f00d40fc6c0 msgr2=0x7f00d40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 shutdown_connections 2024-09-16T01:26:44.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:26:42.877+0000 7f00da6a6640 1 -- 172.21.15.78:0/3830368458 wait complete. 2024-09-16T01:26:44.389 DEBUG:tasks.check_counter:Getting stats from a 2024-09-16T01:26:44.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-16T01:26:44.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:43 smithi078 ceph-mon[30937]: pgmap v501: 129 pgs: 129 active+clean; 3.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 4.8 MiB/s wr, 519 op/s 2024-09-16T01:26:44.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2963754617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/317549825' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-16T01:26:44.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:43 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/3830368458' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-16T01:26:44.647 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 308, 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 297, 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1611616, 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 182960, 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-09-16T01:26:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.051744699, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.020332312, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.042939106, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 296, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.025758434, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000087021 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1611616, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 182960 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 7, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 230, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3552, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 276880, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.019111595, 2024-09-16T01:26:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.011963556, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.004330418, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 228, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.027019650, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000118507 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 276880 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000325536, 2024-09-16T01:26:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000220924, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T01:26:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148771, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 32, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 311, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1568153, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 311, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 311, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1568153, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:45.270 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:45.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:45 smithi078 ceph-mon[30937]: pgmap v502: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.6 MiB/s wr, 392 op/s 2024-09-16T01:26:45.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:45 smithi193 ceph-mon[35701]: pgmap v502: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.6 MiB/s wr, 392 op/s 2024-09-16T01:26:45.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:45 smithi148 ceph-mon[35030]: pgmap v502: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 3.6 MiB/s wr, 392 op/s 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-09-16T01:26:46.092 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-09-16T01:26:46.092 DEBUG:tasks.check_counter:Getting stats from d 2024-09-16T01:26:46.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-16T01:26:46.343 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:46.863 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 317, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 304, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1616048, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 186608, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.060006013, 2024-09-16T01:26:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.019492836, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.053188822, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 304, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.023972492, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000078856 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1616048, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 186608 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 3, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 232, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 480, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 277136, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.017941817, 2024-09-16T01:26:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.012667827, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.002439242, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 229, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.024130309, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000105372 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 277136 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000404323, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T01:26:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000302864, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T01:26:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148714, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 30, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-09-16T01:26:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-09-16T01:26:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 316, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1568803, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 316, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 316, 2024-09-16T01:26:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1568803, 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:46.877 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:47.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:47 smithi078 ceph-mon[30937]: pgmap v503: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-09-16T01:26:47.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:47 smithi193 ceph-mon[35701]: pgmap v503: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-16T01:26:47.581 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-16T01:26:47.581 DEBUG:tasks.check_counter:Getting stats from g 2024-09-16T01:26:47.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-16T01:26:47.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:47 smithi148 ceph-mon[35030]: pgmap v503: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 2.7 MiB/s wr, 301 op/s 2024-09-16T01:26:47.840 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 4, 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 4, 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3072, 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 448, 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-09-16T01:26:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.003096014, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000043112, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.002186407, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 3, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.004034068, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.001344689 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 3, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 230, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 480, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 276784, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.018641403, 2024-09-16T01:26:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.013348339, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.002189778, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 227, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.023952705, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000105518 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 276784 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 310, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 299, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1612160, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 184240, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-09-16T01:26:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.054868462, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.020303648, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.047122816, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 298, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.032124736, 2024-09-16T01:26:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000107801 2024-09-16T01:26:48.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.415 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:48.415 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1612160, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 184240 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148754, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-09-16T01:26:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 313, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1568413, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 313, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 313, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1568413, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:48.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:49.155 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-09-16T01:26:49.156 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-09-16T01:26:49.156 DEBUG:tasks.check_counter:Getting stats from j 2024-09-16T01:26:49.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-16T01:26:49.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:49 smithi078 ceph-mon[30937]: pgmap v504: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-16T01:26:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:26:49.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:49 smithi193 ceph-mon[35701]: pgmap v504: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-16T01:26:49.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:49 smithi148 ceph-mon[35030]: pgmap v504: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 1.5 MiB/s wr, 180 op/s 2024-09-16T01:26:49.916 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 315, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 304, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1613520, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 187440, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.051503994, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.020396303, 2024-09-16T01:26:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.042906065, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 303, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.026576204, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000087710 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1613520, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 187440 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 7, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 235, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3552, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 277760, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.019967089, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.012572641, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.004546210, 2024-09-16T01:26:49.919 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 233, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.026701043, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000114596 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 277760 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.920 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000322760, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000233153, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:49.921 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148731, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-09-16T01:26:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 318, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1569063, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 318, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 318, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1569063, 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-09-16T01:26:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-09-16T01:26:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-09-16T01:26:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-09-16T01:26:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-16T01:26:49.930 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-16T01:26:50.625 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-16T01:26:50.626 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-16T01:26:50.626 DEBUG:tasks.check_counter:Getting stats from b 2024-09-16T01:26:50.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-16T01:26:50.963 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:26:51.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:51 smithi148 ceph-mon[35030]: pgmap v505: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 783 KiB/s wr, 92 op/s 2024-09-16T01:26:51.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:51 smithi078 ceph-mon[30937]: pgmap v505: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 783 KiB/s wr, 92 op/s 2024-09-16T01:26:51.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:51 smithi193 ceph-mon[35701]: pgmap v505: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 783 KiB/s wr, 92 op/s 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 3090, 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 2208, 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 1285980, 2024-09-16T01:26:51.665 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 6701580, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 11, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 4, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.326337387, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.095982358, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.134903256, 2024-09-16T01:26:51.666 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.089127243, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2210, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.195261502, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000088353 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3531, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000719885, 2024-09-16T01:26:51.667 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000203 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 556624, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 165280 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 6763, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 4314, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 2287245, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 5190431, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 217, 2024-09-16T01:26:51.668 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 7, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.430783757, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.159260152, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.325200397, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.015365091, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 4314, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.213999938, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000049605 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 9572, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001403046, 2024-09-16T01:26:51.669 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000146 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 42864, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 574688 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 5020, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 6436, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 3467228, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 4362832, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 18, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 8, 2024-09-16T01:26:51.670 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.467932608, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.163353923, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.265175418, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.083311491, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 6429, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.613201683, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000095380 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 8739, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001283272, 2024-09-16T01:26:51.671 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000146 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 3287648, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 2989696 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:51.672 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1807, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.582933961, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000322597 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 20, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.032243497, 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001612174 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.673 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 1810, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 1575, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1575, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 2.252956128, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001430448 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 235, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 0, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-16T01:26:51.674 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 385, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 3, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 106, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 56, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 39, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 11, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 23, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 856, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 9, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 9, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 4, 2024-09-16T01:26:51.675 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 2163, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 1465, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 234, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 23, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 1, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 1, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 0, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 10, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 905, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 0, 2024-09-16T01:26:51.676 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 2, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 1019, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 1553375011, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 6, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:26:51.677 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 2, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 498, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 1342, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:26:51.678 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 1764, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 1239, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 12, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 75 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 36, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 107, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 64, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 72, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 57, 2024-09-16T01:26:51.679 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 9, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 8, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 1, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-16T01:26:51.680 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 15, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 1319, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 10, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 3, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-16T01:26:51.681 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 1432, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 1430, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 1430, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 2, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 2, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 75, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 74, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 74, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 1, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 1, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 1, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 9679573, 2024-09-16T01:26:51.682 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 9722593, 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 0, 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 372, 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 2.267610252, 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.006095726 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 0 2024-09-16T01:26:51.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 110, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 915, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 805, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 86, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 151, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 65, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 106, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 1177, 2024-09-16T01:26:51.684 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 1071, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 9, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 903, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 894, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 64936, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 231712 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 1976, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 2, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 1810, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 54, 2024-09-16T01:26:51.685 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 71, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 841, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.879826088, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001046166 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 179, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.040417648, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000225796 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.686 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001375144, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001375144 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 98, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.883061433, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.009010830 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.687 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.688 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.689 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-16T01:26:51.690 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 16, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.002340959, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000146309 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 33, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.034532877, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001046450 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.691 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 16, 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.069048329, 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.004315520 2024-09-16T01:26:51.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 337, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.049171885, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000145910 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 18, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.044162633, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.002453479 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.693 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 9, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.072964364, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.008107151 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.694 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 10, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.114696255, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.011469625 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-16T01:26:51.695 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001598519, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000799259 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.696 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.697 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 9, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.022369845, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.002485538 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.698 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 6, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.037390149, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.006231691 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:26:51.699 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 2, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 7, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 5, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 2, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-16T01:26:51.700 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 35, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 17, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 955, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 360, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 360, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:51.701 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:51.702 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:51.703 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 298549, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 202, 2024-09-16T01:26:51.704 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 9704, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 196, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:51.705 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 510561, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 4800, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 1719, 2024-09-16T01:26:51.706 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 7361149, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 37, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 1719, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1719, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 31.106222782, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.018095533 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1719, 2024-09-16T01:26:51.707 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 2543 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 1719, 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 1, 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 1718, 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-16T01:26:51.708 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 338, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 30, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 1, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 373, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 252, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 3, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 530, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-16T01:26:51.709 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 56, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-16T01:26:51.710 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-16T01:26:51.711 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 115, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 32, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 12, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 12, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 0, 2024-09-16T01:26:51.712 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 931, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 0, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 29 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 1, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-16T01:26:51.713 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 981 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 16, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 8, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 16, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 8, 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-16T01:26:51.714 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 14869, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 5415473, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 14869, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 14869, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 5415473, 2024-09-16T01:26:51.715 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:51.716 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 1719, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 7362021, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 1719, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 7362021, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-16T01:26:51.717 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 1719, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 1719, 2024-09-16T01:26:51.718 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 1719, 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 1719, 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.719 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 8, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 872, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 8, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 1, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 872, 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:51.720 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x56176aa20960": { 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 1432, 2024-09-16T01:26:51.721 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 5528289, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 1432, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 372, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 5528289, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:51.722 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:51.723 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds.exported=10 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=36 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=107 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=16 2024-09-16T01:26:52.636 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=10 2024-09-16T01:26:52.636 DEBUG:tasks.check_counter:Getting stats from e 2024-09-16T01:26:52.637 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-16T01:26:52.888 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:26:53.326 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:53 smithi148 ceph-mon[35030]: pgmap v506: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 373 KiB/s wr, 52 op/s 2024-09-16T01:26:53.430 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 619, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 600, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 5952103, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 276802, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 7, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 2, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.082228827, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.026455100, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.060911592, 2024-09-16T01:26:53.431 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.008551751, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 599, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.063270431, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000105626 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 1683376, 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 192208 2024-09-16T01:26:53.432 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 1027, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 1257, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 17045345, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 886915, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 8, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 6, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.111385654, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.031955161, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.039705351, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.029528222, 2024-09-16T01:26:53.433 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1254, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.079214935, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000063169 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 560960 2024-09-16T01:26:53.434 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 2104, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 2104, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 23326428, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 668306, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 5, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.219248011, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.060480390, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.069086545, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.072959348, 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:53.435 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2103, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.112663125, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000053572 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-16T01:26:53.436 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5756, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.158873398, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000027601 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-16T01:26:53.437 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 0, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 0, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.438 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 2367, 2024-09-16T01:26:53.439 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 30, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 2337, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 2325, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 4, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:26:53.440 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 0, 2024-09-16T01:26:53.441 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:26:53.442 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 0 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 0, 2024-09-16T01:26:53.443 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 0, 2024-09-16T01:26:53.444 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 0, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 6561, 2024-09-16T01:26:53.445 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 14, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 2, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 0, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 4194304, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 47871057, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 47847097, 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-16T01:26:53.446 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 6561 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 1741, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 1977, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 236, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 94, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 101, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 7, 2024-09-16T01:26:53.447 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 2367, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 2372, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 5, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 57888, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 215328 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 0, 2024-09-16T01:26:53.448 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 0, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 0, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-16T01:26:53.449 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.450 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.451 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.452 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-16T01:26:53.453 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.453 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.453 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.453 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.454 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-16T01:26:53.455 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-16T01:26:53.457 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.458 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.460 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.461 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 2, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 5, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 3, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 4, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 0, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 0, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 3490, 2024-09-16T01:26:53.462 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:53.463 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:53.464 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:53.465 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 296568, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 2033, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 9704, 2024-09-16T01:26:53.466 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 196, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 6116910, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 86719, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.467 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 3391, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 21, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 3, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 3391, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3391, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 23.527426217, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.006938197 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-16T01:26:53.468 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3391, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 3392 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 3391, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 3391, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 2004, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 1385, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 0, 2024-09-16T01:26:53.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 1, 2024-09-16T01:26:53.470 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-16T01:26:53.471 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-16T01:26:53.472 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 115, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 26, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 9, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 10, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 1, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 0, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 2, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 0 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 0, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:26:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 0 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 0, 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-16T01:26:53.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 3746, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 45965602, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 3746, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3746, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 45965602, 2024-09-16T01:26:53.475 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:53.476 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3391, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 43689482, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3391, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 43689482, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-16T01:26:53.477 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3391, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 3391, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3391, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 3391, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.478 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-16T01:26:53.479 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x55f5b70a6b40": { 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:53.480 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:53.481 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T01:26:53.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:53 smithi078 ceph-mon[30937]: pgmap v506: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 373 KiB/s wr, 52 op/s 2024-09-16T01:26:53.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:53 smithi193 ceph-mon[35701]: pgmap v506: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 373 KiB/s wr, 52 op/s 2024-09-16T01:26:54.273 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-16T01:26:54.273 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-16T01:26:54.273 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-09-16T01:26:54.273 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-09-16T01:26:54.273 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-09-16T01:26:54.274 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-09-16T01:26:54.274 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-09-16T01:26:54.274 DEBUG:tasks.check_counter:Getting stats from h 2024-09-16T01:26:54.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-16T01:26:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 11335, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 15713, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 5557828, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 21332594, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 11, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 4, 2024-09-16T01:26:55.043 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.899505883, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.407280031, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.591552387, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.039256925, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 15712, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 1.696154551, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000107952 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 19165, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.003303637, 2024-09-16T01:26:55.044 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000172 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 5200256, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 8312816 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 3000, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 4520, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 703225, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 18170580, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 18, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 8, 2024-09-16T01:26:55.045 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.281004919, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.117434746, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.097456413, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.053284636, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 4517, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.346567328, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000076725 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3384, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001085048, 2024-09-16T01:26:55.046 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000320 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 1408, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 652016 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 4769, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 7362, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 1966486, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 15432347, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 9, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 7, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.458736125, 2024-09-16T01:26:55.047 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.233811472, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.186404377, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.043414332, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 7361, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.627400406, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000085233 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 9207, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.002703864, 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000293 2024-09-16T01:26:55.048 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 6448, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 5936 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:55.049 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5243, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 5.102218912, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000973148 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 802, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.040577266, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000050595 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.050 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 3348, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 3335, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3335, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 38.283344535, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.011479263 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 13, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 0, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 45, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 4, 2024-09-16T01:26:55.051 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 2020, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 145, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 1875, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 0, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 1812, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 1, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 1045, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 1046, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 7, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 6943, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 5889, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 4, 2024-09-16T01:26:55.052 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 9, 2024-09-16T01:26:55.053 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 0, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 319, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 0, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 1, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 1, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 10, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 905, 2024-09-16T01:26:55.054 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 4, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 234, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:26:55.055 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 0, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 48, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 2676, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 3925, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 5682, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 2018, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 155, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 1462 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.056 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 65, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 38, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 23, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 27, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 111, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 1, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:26:55.057 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 391, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 236, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 155, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 15, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 15667, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 1, 2024-09-16T01:26:55.058 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 4, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 6562, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 6562, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 0, 2024-09-16T01:26:55.059 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 14, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 0, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 0, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 14, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 2, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 0, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 4194304, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 47881819, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 0, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3326, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 35.070126301, 2024-09-16T01:26:55.060 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.010544235 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 0 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 1742, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 1978, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 236, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 95, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 103, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 8, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 2020, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 2406, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 386, 2024-09-16T01:26:55.061 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 1046, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 2818, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 1772, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 72188, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 223520 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 4608, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 69, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 3348, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 360, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 140, 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-16T01:26:55.062 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 435, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 25.423250281, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.058444253 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 86, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.871635312, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.010135294 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.063 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 155, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.445112040, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.002871690 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1410, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 8.694905673, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.006166599 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.064 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000318043, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000159021 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.065 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.066 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 4, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.064017654, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.016004413 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.067 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 571, 2024-09-16T01:26:55.068 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.471543465, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000825820 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 117, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.096282899, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000822930 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 372, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 1.843660236, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.004956075 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.069 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.015560086, 2024-09-16T01:26:55.070 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.007780043 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.071 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.072 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.073 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 181, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.357058846, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001972700 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-16T01:26:55.074 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 2, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 5, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 3, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 2, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 607, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 303, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 959, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:55.075 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:55.076 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:55.077 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 810672, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 1336, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:55.078 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 9704, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 196, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-16T01:26:55.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 6294108, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 92906, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 4502, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 44326343, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 28, 2024-09-16T01:26:55.080 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 4502, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 4502, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 59.123549492, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.013132729 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 4502, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 4885 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 4502, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 96, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 4406, 2024-09-16T01:26:55.081 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 10, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 96, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 3422, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 137, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 1, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 2, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 509, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-16T01:26:55.082 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 20, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-16T01:26:55.083 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-16T01:26:55.084 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 115, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 26, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 12, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 12, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 560, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 0, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 128 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.085 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 1, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 12230 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 24, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 8, 2024-09-16T01:26:55.086 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 486, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 236, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 19100, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 6021592, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 19100, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:55.087 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 19100, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 6021592, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:55.088 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 4502, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 44353179, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 4502, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 44353179, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.089 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 4502, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 4502, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 4502, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 4502, 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:55.090 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 236, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 26836, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 236, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:55.091 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 96, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 26836, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x5619c8c28960": { 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 10762, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 6562, 2024-09-16T01:26:55.092 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 43687515, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 6562, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3326, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 43676753, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:55.093 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T01:26:55.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:55 smithi078 ceph-mon[30937]: pgmap v507: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-09-16T01:26:55.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:55 smithi193 ceph-mon[35701]: pgmap v507: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-09-16T01:26:55.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:55 smithi148 ceph-mon[35030]: pgmap v507: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds.exported=1 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds.imported=10 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=65 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=38 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-16T01:26:55.850 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-16T01:26:55.850 DEBUG:tasks.check_counter:Getting stats from k 2024-09-16T01:26:55.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-16T01:26:56.102 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.b/config 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 0, 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 2, 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 0, 2024-09-16T01:26:56.725 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 0, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 0, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.001113991, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.000000000, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.000258644, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:56.726 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:56.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 3, 2024-09-16T01:26:56.728 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 235, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 480, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 277664, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.013379272, 2024-09-16T01:26:56.729 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.008780120, 2024-09-16T01:26:56.730 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.002085151, 2024-09-16T01:26:56.730 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:56.730 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:56.730 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 232, 2024-09-16T01:26:56.730 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.033469929, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000144266 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-16T01:26:56.731 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 277664 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 321, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 308, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 1617136, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 189168, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 2, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 1, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.057517150, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.018944438, 2024-09-16T01:26:56.732 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.052289497, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 308, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.027990290, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000090877 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 1617136, 2024-09-16T01:26:56.733 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 189168 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 0, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:56.734 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:56.735 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:56.736 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 148731, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 31, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:56.737 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:56.738 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 928, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 2, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-16T01:26:56.739 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 320, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 1569323, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 320, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 320, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 1569323, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-16T01:26:56.740 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-16T01:26:56.741 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-16T01:26:56.741 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:56.741 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T01:26:56.741 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T01:26:57.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:57 smithi148 ceph-mon[35030]: pgmap v508: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 12 op/s 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-09-16T01:26:57.434 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-09-16T01:26:57.434 DEBUG:tasks.check_counter:Getting stats from c 2024-09-16T01:26:57.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-16T01:26:57.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:57 smithi193 ceph-mon[35701]: pgmap v508: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 12 op/s 2024-09-16T01:26:57.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:57 smithi078 ceph-mon[30937]: pgmap v508: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 12 op/s 2024-09-16T01:26:57.757 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:26:58.390 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1925, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1927, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4775945, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 612160, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.169689890, 2024-09-16T01:26:58.391 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.049556573, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.048701436, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.056162043, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1925, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.101334207, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000052641 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.392 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 362, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 590, 2024-09-16T01:26:58.393 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 91640, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 676298, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 8, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.046929482, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.018654837, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.010769432, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.011632198, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 588, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.056303234, 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000095753 2024-09-16T01:26:58.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 563024 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1490, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1471, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3325809, 2024-09-16T01:26:58.395 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 553617, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 3, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.153439578, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.053635241, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.072580926, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.033802308, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1470, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.072522459, 2024-09-16T01:26:58.397 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000049335 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1683536, 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 192368 2024-09-16T01:26:58.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5672, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.168723770, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000029746 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.399 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-09-16T01:26:58.400 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 23, 2024-09-16T01:26:58.401 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 4, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 19, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 7, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 3, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-09-16T01:26:58.402 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-09-16T01:26:58.403 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 1019, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 1553375138, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:26:58.404 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-09-16T01:26:58.405 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-09-16T01:26:58.406 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-09-16T01:26:58.407 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.408 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 1433, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-09-16T01:26:58.409 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 73, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 9679573, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 9745722, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 9722593, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 1433 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 15, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 878, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 863, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 10, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 60, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 50, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 23, 2024-09-16T01:26:58.411 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 884, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 861, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 51132, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215328 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-09-16T01:26:58.412 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-09-16T01:26:58.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.415 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.416 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-09-16T01:26:58.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.420 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-09-16T01:26:58.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.423 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:26:58.425 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 7, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 5, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 5, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-09-16T01:26:58.426 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 4139, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:26:58.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-09-16T01:26:58.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:26:58.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 190476, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 61, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:26:58.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9704, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 196, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-09-16T01:26:58.431 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 92641, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 470, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3417, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-09-16T01:26:58.432 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3417, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3417, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5.233916908, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001531728 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3417, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3418 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3417, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3417, 2024-09-16T01:26:58.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2134, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1280, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-09-16T01:26:58.434 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:26:58.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-09-16T01:26:58.436 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 115, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 26, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 5, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 6, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 1, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-09-16T01:26:58.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T01:26:58.438 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3773, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 7832644, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:58.439 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3773, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3773, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 7832644, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:26:58.440 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3417, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 5551428, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3417, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 5551428, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.441 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3417, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3417, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3417, 2024-09-16T01:26:58.442 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3417, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:26:58.443 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x562086f42b40": { 2024-09-16T01:26:58.444 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:26:58.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:26:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.446 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:26:58.446 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-09-16T01:26:59.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:26:59 smithi078 ceph-mon[30937]: pgmap v509: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 5 op/s 2024-09-16T01:26:59.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:26:59 smithi193 ceph-mon[35701]: pgmap v509: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 5 op/s 2024-09-16T01:26:59.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:26:59 smithi148 ceph-mon[35030]: pgmap v509: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 5 op/s 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-16T01:26:59.725 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-16T01:26:59.725 DEBUG:tasks.check_counter:Getting stats from f 2024-09-16T01:26:59.725 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-16T01:26:59.981 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 3134, 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 3134, 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 917395, 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 995795, 2024-09-16T01:27:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.252815953, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.074766267, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.072285024, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.088799355, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3133, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.135727537, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000043321 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 356, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 337, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1684204, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 193546, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 3, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.056710007, 2024-09-16T01:27:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.020123048, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.052136116, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000062473, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 336, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.022547021, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000067104 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1683648, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 192848 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 4, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 235, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 748, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 653086, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 8, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.013333006, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.008473139, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002641452, 2024-09-16T01:27:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000020629, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 232, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.019488504, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000084002 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 652768 2024-09-16T01:27:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5211, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.129291952, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000024811 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 10, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 10, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-09-16T01:27:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 10, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 3, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-09-16T01:27:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-09-16T01:27:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:27:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-09-16T01:27:00.705 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-09-16T01:27:00.706 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 28, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-09-16T01:27:00.707 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 4224604, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 4224030, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.708 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 28 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 13, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 13, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 0, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 9, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 9, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 0, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 10, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 10, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 0, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-09-16T01:27:00.709 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 50756, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215328 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-09-16T01:27:00.710 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.711 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.712 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-09-16T01:27:00.713 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.714 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.715 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.716 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.717 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-09-16T01:27:00.718 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.720 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.721 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.722 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 4, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 2, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 3507, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-09-16T01:27:00.723 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:27:00.724 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:27:00.725 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 182336, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 56, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-09-16T01:27:00.726 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9704, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 196, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 49820, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 328, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-09-16T01:27:00.727 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3133, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3133, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3133, 2024-09-16T01:27:00.728 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 7.951874267, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002538102 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3133, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3134 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3133, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3133, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-09-16T01:27:00.729 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2038, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1093, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-09-16T01:27:00.730 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-09-16T01:27:00.731 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-09-16T01:27:00.732 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 115, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 26, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 4, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 5, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 1, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-09-16T01:27:00.733 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-09-16T01:27:00.734 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3490, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 2267018, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3490, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3490, 2024-09-16T01:27:00.735 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2267018, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:00.736 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3133, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 30305, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3133, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 30305, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-09-16T01:27:00.737 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3133, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3133, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3133, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3133, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.738 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:00.739 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x5650fe180b40": { 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:00.740 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:00.741 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-09-16T01:27:01.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:01 smithi078 ceph-mon[30937]: pgmap v510: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-09-16T01:27:01.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:01 smithi193 ceph-mon[35701]: pgmap v510: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-09-16T01:27:01.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:01 smithi148 ceph-mon[35030]: pgmap v510: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-09-16T01:27:01.757 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-16T01:27:01.758 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-16T01:27:01.758 DEBUG:tasks.check_counter:Getting stats from i 2024-09-16T01:27:01.758 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-16T01:27:02.016 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1171, 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1088, 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 348525, 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 584014, 2024-09-16T01:27:02.587 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 10, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.099163149, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.036068180, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.061221121, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.006739020, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1087, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.060974925, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000056094 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:02.588 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2130, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000451908, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000212 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 5280, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 10160 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2802, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1682, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2642269, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 935056, 2024-09-16T01:27:02.589 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 15, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.166038133, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.062103175, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.148728469, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000232182, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1681, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.075891182, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045146 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:02.590 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3462, 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000454082, 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000131 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1697536, 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 634848 2024-09-16T01:27:02.591 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.592 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:27:02.592 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 297, 2024-09-16T01:27:02.592 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 409, 2024-09-16T01:27:02.592 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 94089, 2024-09-16T01:27:02.592 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 233949, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 13, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.040648971, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.013979568, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.028451359, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000818261, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 406, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.017539201, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000043200 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.593 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 578, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000105849, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000183 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 87648, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 217488 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.594 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 104, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.005436429, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000052273 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000012669, 2024-09-16T01:27:02.595 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000006334 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 194, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 16, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 16, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.004540972, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000283810 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-09-16T01:27:02.596 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 178, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 11, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 37, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 9, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 28, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 24, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 4, 2024-09-16T01:27:02.597 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 6, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 6, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 5, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 304, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 102, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 178, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 24, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-09-16T01:27:02.598 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 1, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 1, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 4, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 234, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-09-16T01:27:02.599 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 36, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 36, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 56, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 4, 2024-09-16T01:27:02.600 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 3, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 3, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 115, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 69, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 72, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 3, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-09-16T01:27:02.601 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 15, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 359, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-09-16T01:27:02.602 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 28, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 28, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-09-16T01:27:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 2, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 4224604, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 23, 2024-09-16T01:27:02.604 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.673186211, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.029268965 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 41, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 43, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 2, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 86, 2024-09-16T01:27:02.605 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 99, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 13, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 37, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 41, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 4, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 6, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 14, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 8, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 53572, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 223520 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 194, 2024-09-16T01:27:02.606 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 194, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 33, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-09-16T01:27:02.607 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3, 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000900884, 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000300294 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.608 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 7, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.001856042, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000265148 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-09-16T01:27:02.609 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.610 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.611 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.612 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.612 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-09-16T01:27:02.612 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.613 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.001499841, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000299968 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000284205, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000284205 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-09-16T01:27:02.614 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.616 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.618 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-09-16T01:27:02.620 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 4, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 2, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 2, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 7, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 3, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 954, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.623 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:27:02.624 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:27:02.625 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:27:02.626 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 264384, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 113, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9704, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 196, 2024-09-16T01:27:02.627 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 243579, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 1234, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-09-16T01:27:02.628 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 79, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 43390, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 33, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 79, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 79, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.159981636, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.014683311 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 79, 2024-09-16T01:27:02.629 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 118 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 79, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 0, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 79, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 0, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 23, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 13, 2024-09-16T01:27:02.630 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 6, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-09-16T01:27:02.631 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-09-16T01:27:02.632 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 115, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 26, 2024-09-16T01:27:02.633 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 9, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 14, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 5, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 40, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 0, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 6 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-09-16T01:27:02.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 299 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.635 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 4266, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 2664608, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 4266, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4266, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2664608, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:02.637 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 79, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 43390, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 79, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 43390, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-09-16T01:27:02.638 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 79, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 79, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 79, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 79, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:02.640 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x56184def2960": { 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 28, 2024-09-16T01:27:02.641 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 30300, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 28, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 23, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 30300, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:02.642 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-09-16T01:27:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:03 smithi193 ceph-mon[35701]: pgmap v511: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-09-16T01:27:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:03 smithi078 ceph-mon[30937]: pgmap v511: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=3 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=115 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-09-16T01:27:03.542 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-09-16T01:27:03.543 DEBUG:tasks.check_counter:Getting stats from l 2024-09-16T01:27:03.543 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-16T01:27:03.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:03 smithi148 ceph-mon[35030]: pgmap v511: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-09-16T01:27:03.909 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.c/config 2024-09-16T01:27:04.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:04 smithi193 ceph-mon[35701]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T01:27:04.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:04 smithi078 ceph-mon[30937]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T01:27:04.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:04 smithi148 ceph-mon[35030]: osdmap e116: 12 total, 12 up, 12 in 2024-09-16T01:27:04.850 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-09-16T01:27:04.850 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-09-16T01:27:04.850 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 4, 2024-09-16T01:27:04.850 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 4, 2024-09-16T01:27:04.850 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3072, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 448, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 0, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.002998727, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000033976, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002174698, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.004088289, 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001362763 2024-09-16T01:27:04.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 448 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-09-16T01:27:04.852 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 320, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 309, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1614880, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 190640, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 2, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.049078412, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.018784006, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.044579335, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:04.853 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 308, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.017250746, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000056008 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1614880, 2024-09-16T01:27:04.854 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 190640 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 3, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 238, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 480, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 278192, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.013119750, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.008913721, 2024-09-16T01:27:04.855 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.001534490, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 235, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.030204372, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000128529 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:04.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 480, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 278192 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 0, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-09-16T01:27:04.857 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-09-16T01:27:04.858 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-09-16T01:27:04.859 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-09-16T01:27:04.860 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 147876, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 27, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-09-16T01:27:04.861 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 928, 2024-09-16T01:27:04.862 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 2, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 323, 2024-09-16T01:27:04.863 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1569713, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 323, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 323, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1569713, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-09-16T01:27:04.864 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-09-16T01:27:04.865 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:04.865 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-09-16T01:27:04.865 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-09-16T01:27:05.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:05 smithi078 ceph-mon[30937]: pgmap v513: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 39 KiB/s wr, 10 op/s 2024-09-16T01:27:05.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:05 smithi193 ceph-mon[35701]: pgmap v513: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 39 KiB/s wr, 10 op/s 2024-09-16T01:27:05.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:05 smithi148 ceph-mon[35030]: pgmap v513: 129 pgs: 129 active+clean; 3.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 39 KiB/s wr, 10 op/s 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-09-16T01:27:05.673 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-09-16T01:27:05.674 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-09-16T01:27:05.674 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-16T01:27:05.686 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-16T01:27:05.687 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:27:05.687 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:27:05.714 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-16T01:27:05.714 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:27:05.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-16T01:27:06.073 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- 172.21.15.78:0/1520421637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd781082e0 msgr2=0x7fdd7810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 --2- 172.21.15.78:0/1520421637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd781082e0 0x7fdd7810ec50 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009cd0 tx=0x7fdd7002a8e0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- 172.21.15.78:0/1520421637 shutdown_connections 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 --2- 172.21.15.78:0/1520421637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd781082e0 0x7fdd7810ec50 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 --2- 172.21.15.78:0/1520421637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd78103760 0x7fdd78107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 --2- 172.21.15.78:0/1520421637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- 172.21.15.78:0/1520421637 >> 172.21.15.78:0/1520421637 conn(0x7fdd780fc7d0 msgr2=0x7fdd780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:06.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- 172.21.15.78:0/1520421637 shutdown_connections 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- 172.21.15.78:0/1520421637 wait complete. 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 Processor -- start 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.585+0000 7fdd805e3640 1 -- start start 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 0x7fdd7819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd781120b0 con 0x7fdd78103760 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdd78111f30 con 0x7fdd78102db0 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdd78112230 con 0x7fdd781082e0 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34428/0 (socket says 172.21.15.78:34428) 2024-09-16T01:27:06.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 -- 172.21.15.78:0/2977430704 learned_addr learned my addr 172.21.15.78:0/2977430704 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7eb80640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 0x7fdd7819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7e37f640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 -- 172.21.15.78:0/2977430704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 msgr2=0x7fdd7819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 0x7fdd7819ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 -- 172.21.15.78:0/2977430704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 msgr2=0x7fdd78198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 -- 172.21.15.78:0/2977430704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7819d510 con 0x7fdd78103760 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7eb80640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 0x7fdd7819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7e37f640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd7db7e640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d750 tx=0x7fdd7400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:06.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74017070 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.586+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7819d800 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.587+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd74005500 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.587+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74004e00 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.587+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd781a50e0 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.588+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd78103190 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.589+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fdd74004fa0 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.589+0000 7fdd6b7fe640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 0x7fdd640a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.590+0000 7fdd7e37f640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 0x7fdd640a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.590+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9613+0+0 (secure 0 0 0) 0x7fdd740cfb60 con 0x7fdd78103760 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.590+0000 7fdd7e37f640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 0x7fdd640a5870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00abe0 tx=0x7fdd6c00a5c0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.594+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd7409c100 con 0x7fdd78103760 2024-09-16T01:27:06.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.780+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdd7810a2c0 con 0x7fdd78103760 2024-09-16T01:27:06.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.780+0000 7fdd6b7fe640 1 -- 172.21.15.78:0/2977430704 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v116) ==== 81+0+942 (secure 0 0 0) 0x7fdd7409ba00 con 0x7fdd78103760 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/65187856 2024-09-17T01:01:12.814641+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1014137394 2024-09-17T01:01:12.814641+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1698663383 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3690753154 2024-09-17T01:01:12.814641+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1816200719 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1698663383 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/557237582 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.782 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2021750195 2024-09-17T01:01:37.912173+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3073396374 2024-09-17T01:01:12.814641+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4128193457 2024-09-17T01:01:37.912173+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1671709190 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2425087015 2024-09-17T01:01:37.912173+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/238352864 2024-09-17T01:01:37.912173+0000 2024-09-16T01:27:06.783 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3382939699 2024-09-17T01:10:47.191866+0000 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:listed 16 entries 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 msgr2=0x7fdd640a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 0x7fdd640a5870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00abe0 tx=0x7fdd6c00a5c0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 msgr2=0x7fdd78198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d750 tx=0x7fdd7400dc10 comp rx=0 tx=0).stop 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 shutdown_connections 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fdd640a33b0 0x7fdd640a5870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd781082e0 0x7fdd7819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd78103760 0x7fdd78198aa0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 --2- 172.21.15.78:0/2977430704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd78102db0 0x7fdd78198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:06.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.782+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 >> 172.21.15.78:0/2977430704 conn(0x7fdd780fc7d0 msgr2=0x7fdd78106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:06.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.783+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 shutdown_connections 2024-09-16T01:27:06.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:06.783+0000 7fdd805e3640 1 -- 172.21.15.78:0/2977430704 wait complete. 2024-09-16T01:27:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:07 smithi078 ceph-mon[30937]: pgmap v514: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:07.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:07 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2977430704' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:27:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:07 smithi193 ceph-mon[35701]: pgmap v514: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:07.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:07 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2977430704' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:27:07.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:07 smithi148 ceph-mon[35030]: pgmap v514: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:07.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:07 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2977430704' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-16T01:27:08.513 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:27:08.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:27:08.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /tmp/cephfs 2024-09-16T01:27:08.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:27:08.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:27:08.657 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-09-16T01:27:08.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:27:08.694 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-09-16T01:27:08.782 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-09-16T01:27:08.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:27:08.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-09-16T01:27:08.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:27:08.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-09-16T01:27:08.914 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-16T01:27:08.925 INFO:tasks.cephadm:Teardown begin 2024-09-16T01:27:08.925 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:27:08.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:27:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:09.454 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:09 smithi078 ceph-mon[30937]: pgmap v515: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:09.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:09 smithi193 ceph-mon[35701]: pgmap v515: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:09.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:09 smithi148 ceph-mon[35030]: pgmap v515: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 28 op/s 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 -- 172.21.15.78:0/4029287616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 msgr2=0x7f25a810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 --2- 172.21.15.78:0/4029287616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a810bb50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f259c009900 tx=0x7f259c02a480 comp rx=0 tx=0).stop 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 -- 172.21.15.78:0/4029287616 shutdown_connections 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 --2- 172.21.15.78:0/4029287616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a810c170 0x7f25a810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 --2- 172.21.15.78:0/4029287616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a810bb50 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 --2- 172.21.15.78:0/4029287616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25a8102d40 0x7f25a8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 -- 172.21.15.78:0/4029287616 >> 172.21.15.78:0/4029287616 conn(0x7f25a80fc7a0 msgr2=0x7f25a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:09.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 -- 172.21.15.78:0/4029287616 shutdown_connections 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.678+0000 7f25acb49640 1 -- 172.21.15.78:0/4029287616 wait complete. 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 Processor -- start 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 -- start start 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a8102d40 0x7f25a81009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25a810c170 0x7f25a8101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8111fb0 con 0x7f25a8103660 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8111e30 con 0x7f25a8102d40 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25acb49640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8112130 con 0x7f25a810c170 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25a5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25a5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34444/0 (socket says 172.21.15.78:34444) 2024-09-16T01:27:09.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25a5d9b640 1 -- 172.21.15.78:0/4278815597 learned_addr learned my addr 172.21.15.78:0/4278815597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.679+0000 7f25a659c640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a8102d40 0x7f25a81009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 -- 172.21.15.78:0/4278815597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25a810c170 msgr2=0x7f25a8101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25a810c170 0x7f25a8101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 -- 172.21.15.78:0/4278815597 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a8102d40 msgr2=0x7f25a81009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a8102d40 0x7f25a81009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 -- 172.21.15.78:0/4278815597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a81a6970 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25a5d9b640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f259c006010 tx=0x7f259c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c03f030 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f259c003ce0 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a81a6c00 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c035070 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.680+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a81a70e0 con 0x7f25a8103660 2024-09-16T01:27:09.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.681+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a8103120 con 0x7f25a8103660 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.682+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f259c042030 con 0x7f25a8103660 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.683+0000 7f25977fe640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 0x7f25900a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.683+0000 7f25a659c640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 0x7f25900a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.683+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9613+0+0 (secure 0 0 0) 0x7f259c0f0890 con 0x7f25a8103660 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.683+0000 7f25a659c640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 0x7f25900a5870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2598009990 tx=0x7f2598009110 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:09.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.687+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f259c0bce30 con 0x7f25a8103660 2024-09-16T01:27:09.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.888+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a81071e0 con 0x7f25a8103660 2024-09-16T01:27:09.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.888+0000 7f25977fe640 1 -- 172.21.15.78:0/4278815597 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f259c0bc730 con 0x7f25a8103660 2024-09-16T01:27:09.890 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:27:09.890 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:27:09.890 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 msgr2=0x7f25900a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 0x7f25900a5870 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2598009990 tx=0x7f2598009110 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 msgr2=0x7f25a8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f259c006010 tx=0x7f259c0042f0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 shutdown_connections 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f25900a33b0 0x7f25900a5870 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25a810c170 0x7f25a8101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a8103660 0x7f25a8100ee0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 --2- 172.21.15.78:0/4278815597 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a8102d40 0x7f25a81009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.890+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 >> 172.21.15.78:0/4278815597 conn(0x7f25a80fc7a0 msgr2=0x7f25a80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:09.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.891+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 shutdown_connections 2024-09-16T01:27:09.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:09.891+0000 7f25acb49640 1 -- 172.21.15.78:0/4278815597 wait complete. 2024-09-16T01:27:10.331 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:10 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/4278815597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:10.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:10 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/4278815597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:10.616 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-16T01:27:10.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:10 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/4278815597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:10.873 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 -- 172.21.15.78:0/3042607307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 msgr2=0x7f560c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 --2- 172.21.15.78:0/3042607307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c10d950 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009930 tx=0x7f55fc02a420 comp rx=0 tx=0).stop 2024-09-16T01:27:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 -- 172.21.15.78:0/3042607307 shutdown_connections 2024-09-16T01:27:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 --2- 172.21.15.78:0/3042607307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c10d950 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 --2- 172.21.15.78:0/3042607307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c100a80 0x7f560c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 --2- 172.21.15.78:0/3042607307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c106a80 0x7f560c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 -- 172.21.15.78:0/3042607307 >> 172.21.15.78:0/3042607307 conn(0x7f560c0fc7a0 msgr2=0x7f560c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 -- 172.21.15.78:0/3042607307 shutdown_connections 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.330+0000 7f5610aee640 1 -- 172.21.15.78:0/3042607307 wait complete. 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 Processor -- start 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 -- start start 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c100a80 0x7f560c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 0x7f560c19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f560c110ea0 con 0x7f560c101420 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f560c110d20 con 0x7f560c106a80 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5610aee640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f560c111020 con 0x7f560c100a80 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5609d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5609d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34454/0 (socket says 172.21.15.78:34454) 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.331+0000 7f5609d9b640 1 -- 172.21.15.78:0/396463169 learned_addr learned my addr 172.21.15.78:0/396463169 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f560ad9d640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 0x7f560c19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f560a59c640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c100a80 0x7f560c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 -- 172.21.15.78:0/396463169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c100a80 msgr2=0x7f560c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c100a80 0x7f560c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 -- 172.21.15.78:0/396463169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 msgr2=0x7f560c19ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 0x7f560c19ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 -- 172.21.15.78:0/396463169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560c19d3a0 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f560ad9d640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 0x7f560c19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5609d9b640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f55f400c920 tx=0x7f55f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4007c70 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560c19d690 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55f4007e10 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4018890 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.332+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f560c1a4f70 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.333+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560c100ee0 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.334+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f55f4018a70 con 0x7f560c101420 2024-09-16T01:27:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.335+0000 7f56037fe640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 0x7f55f80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:11.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.335+0000 7f560a59c640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 0x7f55f80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:11.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.335+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9525+0+0 (secure 0 0 0) 0x7f55f40d0310 con 0x7f560c101420 2024-09-16T01:27:11.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.335+0000 7f560a59c640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 0x7f55f80a5870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5604007dc0 tx=0x7f56040072f0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:11.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.339+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f55f4014030 con 0x7f560c101420 2024-09-16T01:27:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:11 smithi078 ceph-mon[30937]: pgmap v516: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 157 KiB/s wr, 28 op/s 2024-09-16T01:27:11.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:11 smithi078 ceph-mon[30937]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T01:27:11.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.540+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f560c10a080 con 0x7f560c101420 2024-09-16T01:27:11.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.541+0000 7f56037fe640 1 -- 172.21.15.78:0/396463169 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11918 (secure 0 0 0) 0x7f55f400de20 con 0x7f560c101420 2024-09-16T01:27:11.542 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:27:11.542 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":31,"btime":"2024-09-16T01:14:03:002675+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/557283089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":557283089},{"type":"v1","addr":"172.21.15.78:6835","nonce":557283089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4143414338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4143414338},{"type":"v1","addr":"172.21.15.78:6837","nonce":4143414338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/2440325522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":2440325522},{"type":"v1","addr":"172.21.15.78:6839","nonce":2440325522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14592,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/643835923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":643835923},{"type":"v1","addr":"172.21.15.78:6841","nonce":643835923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/4092784262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":4092784262},{"type":"v1","addr":"172.21.15.193:6833","nonce":4092784262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/909591038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":909591038},{"type":"v1","addr":"172.21.15.148:6833","nonce":909591038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-16T01:08:29.385380+0000","modified":"2024-09-16T01:14:02.782631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24485,"mds_1":24460,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24434":{"gid":24434,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1399158659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1399158659},{"type":"v1","addr":"172.21.15.148:6835","nonce":1399158659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1850679548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1850679548},{"type":"v1","addr":"172.21.15.193:6835","nonce":1850679548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1637418104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1637418104},{"type":"v1","addr":"172.21.15.193:6837","nonce":1637418104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/330388800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":330388800},{"type":"v1","addr":"172.21.15.148:6837","nonce":330388800}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":15,"addr":"172.21.15.193:6839/1136630649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":1136630649},{"type":"v1","addr":"172.21.15.193:6839","nonce":1136630649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6839/3106047695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":3106047695},{"type":"v1","addr":"172.21.15.148:6839","nonce":3106047695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24460,24455]},"id":1}]} 2024-09-16T01:27:11.542 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 31 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 msgr2=0x7f55f80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 0x7f55f80a5870 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5604007dc0 tx=0x7f56040072f0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 msgr2=0x7f560c198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f55f400c920 tx=0x7f55f400cde0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 shutdown_connections 2024-09-16T01:27:11.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f55f80a33b0 0x7f55f80a5870 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f560c106a80 0x7f560c19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f560c101420 0x7f560c198930 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 --2- 172.21.15.78:0/396463169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f560c100a80 0x7f560c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 >> 172.21.15.78:0/396463169 conn(0x7f560c0fc7a0 msgr2=0x7f560c06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 shutdown_connections 2024-09-16T01:27:11.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:11.543+0000 7f5610aee640 1 -- 172.21.15.78:0/396463169 wait complete. 2024-09-16T01:27:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:11 smithi193 ceph-mon[35701]: pgmap v516: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 157 KiB/s wr, 28 op/s 2024-09-16T01:27:11.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:11 smithi193 ceph-mon[35701]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T01:27:11.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:11 smithi148 ceph-mon[35030]: pgmap v516: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 157 KiB/s wr, 28 op/s 2024-09-16T01:27:11.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:11 smithi148 ceph-mon[35030]: osdmap e117: 12 total, 12 up, 12 in 2024-09-16T01:27:12.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-16T01:27:12.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:12 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/396463169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:12.545 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:12.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:12 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/396463169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:12.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:12 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/396463169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 -- 172.21.15.78:0/1692711069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0106f70 msgr2=0x7f7da010eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 --2- 172.21.15.78:0/1692711069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0106f70 0x7f7da010eab0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009ca0 tx=0x7f7d9802a8c0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 -- 172.21.15.78:0/1692711069 shutdown_connections 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 --2- 172.21.15.78:0/1692711069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0106f70 0x7f7da010eab0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 --2- 172.21.15.78:0/1692711069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100eb0 0x7f7da0106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 --2- 172.21.15.78:0/1692711069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0100500 0x7f7da01008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 -- 172.21.15.78:0/1692711069 >> 172.21.15.78:0/1692711069 conn(0x7f7da00fc5c0 msgr2=0x7f7da00fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 -- 172.21.15.78:0/1692711069 shutdown_connections 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.414+0000 7f7da4971640 1 -- 172.21.15.78:0/1692711069 wait complete. 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 Processor -- start 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 -- start start 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100500 0x7f7da01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0106f70 0x7f7da019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0112010 con 0x7f7da0100eb0 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0111e90 con 0x7f7da0100500 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7da4971640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7da0112190 con 0x7f7da0106f70 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.415+0000 7f7d9dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:13.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100500 0x7f7da01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34468/0 (socket says 172.21.15.78:34468) 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 -- 172.21.15.78:0/2402585165 learned_addr learned my addr 172.21.15.78:0/2402585165 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9ed9d640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0106f70 0x7f7da019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 -- 172.21.15.78:0/2402585165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0106f70 msgr2=0x7f7da019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0106f70 0x7f7da019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 -- 172.21.15.78:0/2402585165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100500 msgr2=0x7f7da01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100500 0x7f7da01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 -- 172.21.15.78:0/2402585165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da019d460 con 0x7f7da0100eb0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d9dd9b640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c8c0 tx=0x7f7d9400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94007bb0 con 0x7f7da0100eb0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d94007d50 con 0x7f7da0100eb0 2024-09-16T01:27:13.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94018790 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da019d750 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.416+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da01a5030 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.417+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da01008e0 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.418+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f7d94018950 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.419+0000 7f7d8f7fe640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 0x7f7d880a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.419+0000 7f7d9e59c640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 0x7f7d880a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.419+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9525+0+0 (secure 0 0 0) 0x7f7d940d0620 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.420+0000 7f7d9e59c640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 0x7f7d880a5870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7da0069fc0 tx=0x7f7d90009110 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:13.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.422+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7d94014030 con 0x7f7da0100eb0 2024-09-16T01:27:13.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:13 smithi078 ceph-mon[30937]: pgmap v518: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 155 KiB/s wr, 31 op/s 2024-09-16T01:27:13.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:13 smithi193 ceph-mon[35701]: pgmap v518: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 155 KiB/s wr, 31 op/s 2024-09-16T01:27:13.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.589+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7da0199810 con 0x7f7da0100eb0 2024-09-16T01:27:13.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.591+0000 7f7d8f7fe640 1 -- 172.21.15.78:0/2402585165 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25409 (secure 0 0 0) 0x7f7d9409cc40 con 0x7f7da0100eb0 2024-09-16T01:27:13.593 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-16T01:27:13.593 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":117,"fsid":"fa4731de-73c6-11ef-bceb-c7b262605968","created":"2024-09-16T01:00:45.346692+0000","modified":"2024-09-16T01:27:10.635188+0000","last_up_change":"2024-09-16T01:07:36.065581+0000","last_in_change":"2024-09-16T01:07:13.468244+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:03:52.230028+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:08:25.352496+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:08:27.537592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-16T01:08:33.982278+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ded124f4-818a-4255-a9cb-dcaa0fc71556","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6803","nonce":3172670933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6805","nonce":3172670933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6809","nonce":3172670933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3172670933},{"type":"v1","addr":"172.21.15.78:6807","nonce":3172670933}]},"public_addr":"172.21.15.78:6803/3172670933","cluster_addr":"172.21.15.78:6805/3172670933","heartbeat_back_addr":"172.21.15.78:6809/3172670933","heartbeat_front_addr":"172.21.15.78:6807/3172670933","state":["exists","up"]},{"osd":1,"uuid":"82da4924-eaf5-44df-adce-7f605462c130","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6801","nonce":593162683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6803","nonce":593162683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6807","nonce":593162683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":593162683},{"type":"v1","addr":"172.21.15.148:6805","nonce":593162683}]},"public_addr":"172.21.15.148:6801/593162683","cluster_addr":"172.21.15.148:6803/593162683","heartbeat_back_addr":"172.21.15.148:6807/593162683","heartbeat_front_addr":"172.21.15.148:6805/593162683","state":["exists","up"]},{"osd":2,"uuid":"51654cac-6b84-4b81-9861-6218a4f82eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6801","nonce":3453663394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6803","nonce":3453663394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6807","nonce":3453663394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3453663394},{"type":"v1","addr":"172.21.15.193:6805","nonce":3453663394}]},"public_addr":"172.21.15.193:6801/3453663394","cluster_addr":"172.21.15.193:6803/3453663394","heartbeat_back_addr":"172.21.15.193:6807/3453663394","heartbeat_front_addr":"172.21.15.193:6805/3453663394","state":["exists","up"]},{"osd":3,"uuid":"3aa4ba88-9fb8-4dfc-bc12-f313c953351a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6811","nonce":419734598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6813","nonce":419734598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6817","nonce":419734598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":419734598},{"type":"v1","addr":"172.21.15.78:6815","nonce":419734598}]},"public_addr":"172.21.15.78:6811/419734598","cluster_addr":"172.21.15.78:6813/419734598","heartbeat_back_addr":"172.21.15.78:6817/419734598","heartbeat_front_addr":"172.21.15.78:6815/419734598","state":["exists","up"]},{"osd":4,"uuid":"b8411f2d-a262-4113-95f9-ca6ae72ab967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6809","nonce":2237990015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6811","nonce":2237990015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6815","nonce":2237990015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2237990015},{"type":"v1","addr":"172.21.15.148:6813","nonce":2237990015}]},"public_addr":"172.21.15.148:6809/2237990015","cluster_addr":"172.21.15.148:6811/2237990015","heartbeat_back_addr":"172.21.15.148:6815/2237990015","heartbeat_front_addr":"172.21.15.148:6813/2237990015","state":["exists","up"]},{"osd":5,"uuid":"3508afd5-2958-42dd-81c0-5f50be889ec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6809","nonce":2281160557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6811","nonce":2281160557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6815","nonce":2281160557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2281160557},{"type":"v1","addr":"172.21.15.193:6813","nonce":2281160557}]},"public_addr":"172.21.15.193:6809/2281160557","cluster_addr":"172.21.15.193:6811/2281160557","heartbeat_back_addr":"172.21.15.193:6815/2281160557","heartbeat_front_addr":"172.21.15.193:6813/2281160557","state":["exists","up"]},{"osd":6,"uuid":"0c2c983a-999e-45fe-bd9e-270546696ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6819","nonce":3932044692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6821","nonce":3932044692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6825","nonce":3932044692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3932044692},{"type":"v1","addr":"172.21.15.78:6823","nonce":3932044692}]},"public_addr":"172.21.15.78:6819/3932044692","cluster_addr":"172.21.15.78:6821/3932044692","heartbeat_back_addr":"172.21.15.78:6825/3932044692","heartbeat_front_addr":"172.21.15.78:6823/3932044692","state":["exists","up"]},{"osd":7,"uuid":"c9855f37-2fd8-44a6-a6d9-80cf978e357a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6817","nonce":883929363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6819","nonce":883929363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6823","nonce":883929363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":883929363},{"type":"v1","addr":"172.21.15.148:6821","nonce":883929363}]},"public_addr":"172.21.15.148:6817/883929363","cluster_addr":"172.21.15.148:6819/883929363","heartbeat_back_addr":"172.21.15.148:6823/883929363","heartbeat_front_addr":"172.21.15.148:6821/883929363","state":["exists","up"]},{"osd":8,"uuid":"be96be1e-a654-46f9-a1aa-d2ce999d7a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6817","nonce":692862816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6819","nonce":692862816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6823","nonce":692862816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":692862816},{"type":"v1","addr":"172.21.15.193:6821","nonce":692862816}]},"public_addr":"172.21.15.193:6817/692862816","cluster_addr":"172.21.15.193:6819/692862816","heartbeat_back_addr":"172.21.15.193:6823/692862816","heartbeat_front_addr":"172.21.15.193:6821/692862816","state":["exists","up"]},{"osd":9,"uuid":"fc303551-6913-47a4-ade8-be306c49a4a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6827","nonce":409695790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6829","nonce":409695790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6833","nonce":409695790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":409695790},{"type":"v1","addr":"172.21.15.78:6831","nonce":409695790}]},"public_addr":"172.21.15.78:6827/409695790","cluster_addr":"172.21.15.78:6829/409695790","heartbeat_back_addr":"172.21.15.78:6833/409695790","heartbeat_front_addr":"172.21.15.78:6831/409695790","state":["exists","up"]},{"osd":10,"uuid":"ae7b8367-0244-4150-97f6-8ff99149e440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6825","nonce":85363667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6827","nonce":85363667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6831","nonce":85363667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":85363667},{"type":"v1","addr":"172.21.15.148:6829","nonce":85363667}]},"public_addr":"172.21.15.148:6825/85363667","cluster_addr":"172.21.15.148:6827/85363667","heartbeat_back_addr":"172.21.15.148:6831/85363667","heartbeat_front_addr":"172.21.15.148:6829/85363667","state":["exists","up"]},{"osd":11,"uuid":"e9f87f81-392a-4490-9a01-394b494b5251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6825","nonce":1055668594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6827","nonce":1055668594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6831","nonce":1055668594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1055668594},{"type":"v1","addr":"172.21.15.193:6829","nonce":1055668594}]},"public_addr":"172.21.15.193:6825/1055668594","cluster_addr":"172.21.15.193:6827/1055668594","heartbeat_back_addr":"172.21.15.193:6831/1055668594","heartbeat_front_addr":"172.21.15.193:6829/1055668594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:02:59.945747+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:24.529301+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:03:48.766074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:13.851111+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:04:38.900500+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:03.909574+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:29.993493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:05:55.027972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:19.990342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:06:45.585368+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:10.150707+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:07:34.382092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.28","mappings":[{"from":3,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/65187856":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1014137394":"2024-09-17T01:01:12.814641+0000","172.21.15.78:6800/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3690753154":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/1816200719":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/1698663383":"2024-09-17T01:10:47.191866+0000","172.21.15.78:0/557237582":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6800/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/2021750195":"2024-09-17T01:01:37.912173+0000","172.21.15.78:6801/3073396374":"2024-09-17T01:01:12.814641+0000","172.21.15.78:0/4128193457":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/1671709190":"2024-09-17T01:10:47.191866+0000","172.21.15.78:6801/2425087015":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/238352864":"2024-09-17T01:01:37.912173+0000","172.21.15.78:0/3382939699":"2024-09-17T01:10:47.191866+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]},{"pool":4,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:27:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 msgr2=0x7f7d880a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 0x7f7d880a5870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7da0069fc0 tx=0x7f7d90009110 comp rx=0 tx=0).stop 2024-09-16T01:27:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 msgr2=0x7f7da01989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c8c0 tx=0x7f7d9400ccc0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 shutdown_connections 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f7d880a33b0 0x7f7d880a5870 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7da0106f70 0x7f7da019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da0100eb0 0x7f7da01989f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 --2- 172.21.15.78:0/2402585165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7da0100500 0x7f7da01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 >> 172.21.15.78:0/2402585165 conn(0x7f7da00fc5c0 msgr2=0x7f7da0105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 shutdown_connections 2024-09-16T01:27:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:13.593+0000 7f7da4971640 1 -- 172.21.15.78:0/2402585165 wait complete. 2024-09-16T01:27:13.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:13 smithi148 ceph-mon[35030]: pgmap v518: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 155 KiB/s wr, 31 op/s 2024-09-16T01:27:14.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-16T01:27:14.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:14 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2402585165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:27:14.563 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:14.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:14 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2402585165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:27:14.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:14 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2402585165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- 172.21.15.78:0/881702140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 msgr2=0x7f6e081029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/881702140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e081029b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009ca0 tx=0x7f6df802a950 comp rx=0 tx=0).stop 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- 172.21.15.78:0/881702140 shutdown_connections 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/881702140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08102ef0 0x7f6e0810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/881702140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e081029b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/881702140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08100e20 0x7f6e08101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- 172.21.15.78:0/881702140 >> 172.21.15.78:0/881702140 conn(0x7f6e080fc7d0 msgr2=0x7f6e080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- 172.21.15.78:0/881702140 shutdown_connections 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- 172.21.15.78:0/881702140 wait complete. 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 Processor -- start 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.059+0000 7f6e0ec49640 1 -- start start 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08100e20 0x7f6e081985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08102ef0 0x7f6e0819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08110fa0 con 0x7f6e08102570 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08110e20 con 0x7f6e08100e20 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0ec49640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08111120 con 0x7f6e08102ef0 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33586/0 (socket says 172.21.15.78:33586) 2024-09-16T01:27:15.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 -- 172.21.15.78:0/2664319537 learned_addr learned my addr 172.21.15.78:0/2664319537 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0c9e5640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08100e20 0x7f6e081985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e0d1e6640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08102ef0 0x7f6e0819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 -- 172.21.15.78:0/2664319537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08102ef0 msgr2=0x7f6e0819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08102ef0 0x7f6e0819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 -- 172.21.15.78:0/2664319537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08100e20 msgr2=0x7f6e081985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08100e20 0x7f6e081985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 -- 172.21.15.78:0/2664319537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0819d570 con 0x7f6e08102570 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e07fff640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6df8005cc0 tx=0x7f6df802f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8034040 con 0x7f6e08102570 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df8030920 con 0x7f6e08102570 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.060+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df80047c0 con 0x7f6e08102570 2024-09-16T01:27:15.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.061+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0819d800 con 0x7f6e08102570 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.061+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e0819dc90 con 0x7f6e08102570 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.062+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e08101200 con 0x7f6e08102570 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.063+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f6df8037020 con 0x7f6e08102570 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.063+0000 7f6e05ffb640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 0x7f6df40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.063+0000 7f6e0c9e5640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 0x7f6df40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.063+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9525+0+0 (secure 0 0 0) 0x7f6df80f0fd0 con 0x7f6e08102570 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.064+0000 7f6e0c9e5640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 0x7f6df40a58f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009990 tx=0x7f6e00009110 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:15.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.070+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6df80bd6d0 con 0x7f6e08102570 2024-09-16T01:27:15.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.273+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6e0810a0d0 con 0x7f6e08102570 2024-09-16T01:27:15.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.276+0000 7f6e05ffb640 1 -- 172.21.15.78:0/2664319537 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7f6df80bcfd0 con 0x7f6e08102570 2024-09-16T01:27:15.278 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:27:15.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 msgr2=0x7f6df40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:15.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 0x7f6df40a58f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009990 tx=0x7f6e00009110 comp rx=0 tx=0).stop 2024-09-16T01:27:15.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 msgr2=0x7f6e08198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6df8005cc0 tx=0x7f6df802f650 comp rx=0 tx=0).stop 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 shutdown_connections 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f6df40a3430 0x7f6df40a58f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e08102ef0 0x7f6e0819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e08102570 0x7f6e08198b00 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 --2- 172.21.15.78:0/2664319537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e08100e20 0x7f6e081985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 >> 172.21.15.78:0/2664319537 conn(0x7f6e080fc7d0 msgr2=0x7f6e0806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 shutdown_connections 2024-09-16T01:27:15.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:15.279+0000 7f6e0ec49640 1 -- 172.21.15.78:0/2664319537 wait complete. 2024-09-16T01:27:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a[30933]: 2024-09-16T01:27:15.274+0000 7f40b2fc9640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:27:15.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: pgmap v519: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 129 KiB/s wr, 26 op/s 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:27:15.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:15 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T01:27:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: pgmap v519: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 129 KiB/s wr, 26 op/s 2024-09-16T01:27:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:27:15.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.547 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: starting mds.i at 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mds[65934]: -- 172.21.15.193:0/377807808 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5634fd0d2680 con 0x5634fd0a7000 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: starting mds.f at 2024-09-16T01:27:15.548 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mds[66238]: -- 172.21.15.193:0/1945382103 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560ca9590680 con 0x560ca9565400 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-mds[66238]: -- 172.21.15.193:0/1945382103 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ca9587860 con 0x560ca9565400 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: starting mds.c at 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: 2024-09-16T01:27:15.320+0000 7f5f6616c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-09-16T01:27:15.549 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:27:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: 2024-09-16T01:27:15.321+0000 7f5f6616c640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6838/0: (98) Address already in use 2024-09-16T01:27:15.636 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.636 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.636 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mds[64477]: -- 172.21.15.148:0/3995630083 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5600e7ba5c20 con 0x5600e7b83000 2024-09-16T01:27:15.637 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: starting mds.b at 2024-09-16T01:27:15.637 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.637 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.637 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mds[63873]: -- 172.21.15.148:0/1291910007 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558439515860 con 0x5584394f3400 2024-09-16T01:27:15.637 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: starting mds.e at 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: ignoring --setuser ceph since I am not root 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: ignoring --setgroup ceph since I am not root 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mds[64175]: -- 172.21.15.148:0/2144087550 <== mon.2 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558dbaf31860 con 0x558dbaf0f000 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: starting mds.h at 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: pgmap v519: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 129 KiB/s wr, 26 op/s 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-16T01:27:15.638 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' 2024-09-16T01:27:15.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-16T01:27:15.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-16T01:27:15.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:27:15.639 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:15 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2664319537' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-16T01:27:15.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:27:16.230 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:16 smithi193 ceph-mon[35701]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T01:27:16.546 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:16 smithi193 ceph-mon[35701]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T01:27:16.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:16 smithi148 ceph-mon[35030]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T01:27:16.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:16 smithi148 ceph-mon[35030]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T01:27:16.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:16 smithi078 ceph-mon[30937]: osdmap e118: 12 total, 12 up, 12 in 2024-09-16T01:27:16.644 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:16 smithi078 ceph-mon[30937]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-16T01:27:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 -- 172.21.15.78:0/2114323483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 msgr2=0x7fefac10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 --2- 172.21.15.78:0/2114323483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac10eaa0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fefa4009ca0 tx=0x7fefa402a8c0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 -- 172.21.15.78:0/2114323483 shutdown_connections 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 --2- 172.21.15.78:0/2114323483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac10eaa0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 --2- 172.21.15.78:0/2114323483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac1027a0 0x7fefac1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 --2- 172.21.15.78:0/2114323483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac101df0 0x7fefac1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 -- 172.21.15.78:0/2114323483 >> 172.21.15.78:0/2114323483 conn(0x7fefac0fd8b0 msgr2=0x7fefac0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 -- 172.21.15.78:0/2114323483 shutdown_connections 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.724+0000 7fefb2491640 1 -- 172.21.15.78:0/2114323483 wait complete. 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 Processor -- start 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 -- start start 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac101df0 0x7fefac198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac1027a0 0x7fefac1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:16.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fefac111fa0 con 0x7fefac106f20 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fefac111e20 con 0x7fefac1027a0 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb2491640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefac112120 con 0x7fefac101df0 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33610/0 (socket says 172.21.15.78:33610) 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 -- 172.21.15.78:0/87373892 learned_addr learned my addr 172.21.15.78:0/87373892 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefab7fe640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac1027a0 0x7fefac1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefabfff640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac101df0 0x7fefac198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 -- 172.21.15.78:0/87373892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac101df0 msgr2=0x7fefac198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac101df0 0x7fefac198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 -- 172.21.15.78:0/87373892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac1027a0 msgr2=0x7fefac1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac1027a0 0x7fefac1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.725+0000 7fefb0a2e640 1 -- 172.21.15.78:0/87373892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac19d310 con 0x7fefac106f20 2024-09-16T01:27:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefb0a2e640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fefa4031a60 tx=0x7fefa4031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003b70 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefac19d5a0 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa4003d10 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4004770 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.726+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefac1a4ee0 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.727+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefac1021d0 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.728+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7fefa402f790 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.729+0000 7fefa97fa640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 0x7fef980a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.729+0000 7fefabfff640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 0x7fef980a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.729+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9695+0+0 (secure 0 0 0) 0x7fefa40f1610 con 0x7fefac106f20 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.729+0000 7fefabfff640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 0x7fef980a5870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fef9c005e10 tx=0x7fef9c005da0 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.733+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fefa40bdc10 con 0x7fefac106f20 2024-09-16T01:27:16.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:16.935+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fefac1a5110 con 0x7fefac106f20 2024-09-16T01:27:17.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.291+0000 7fefa97fa640 1 -- 172.21.15.78:0/87373892 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) ==== 104+0+0 (secure 0 0 0) 0x7fefa40bd510 con 0x7fefac106f20 2024-09-16T01:27:17.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 msgr2=0x7fef980a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:17.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 0x7fef980a5870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fef9c005e10 tx=0x7fef9c005da0 comp rx=0 tx=0).stop 2024-09-16T01:27:17.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 msgr2=0x7fefac19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:17.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fefa4031a60 tx=0x7fefa4031a90 comp rx=0 tx=0).stop 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 shutdown_connections 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7fef980a33b0 0x7fef980a5870 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fefac106f20 0x7fefac19cc30 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefac1027a0 0x7fefac1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 --2- 172.21.15.78:0/87373892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefac101df0 0x7fefac198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 >> 172.21.15.78:0/87373892 conn(0x7fefac0fd8b0 msgr2=0x7fefac0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 shutdown_connections 2024-09-16T01:27:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:17.293+0000 7fefb2491640 1 -- 172.21.15.78:0/87373892 wait complete. 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: pgmap v521: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 12 op/s 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6834/3612423816,v1:172.21.15.193:6837/3612423816] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6835/3660781575,v1:172.21.15.193:6838/3660781575] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.193:6836/946028959,v1:172.21.15.193:6839/946028959] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6834/4183609759,v1:172.21.15.148:6837/4183609759] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6835/1595305815,v1:172.21.15.148:6838/1595305815] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: mds.? [v2:172.21.15.148:6836/3334163143,v1:172.21.15.148:6839/3334163143] up:boot 2024-09-16T01:27:17.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:27:17.637 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:17 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: pgmap v521: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 12 op/s 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6834/3612423816,v1:172.21.15.193:6837/3612423816] up:boot 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6835/3660781575,v1:172.21.15.193:6838/3660781575] up:boot 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.193:6836/946028959,v1:172.21.15.193:6839/946028959] up:boot 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6834/4183609759,v1:172.21.15.148:6837/4183609759] up:boot 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6835/1595305815,v1:172.21.15.148:6838/1595305815] up:boot 2024-09-16T01:27:17.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: mds.? [v2:172.21.15.148:6836/3334163143,v1:172.21.15.148:6839/3334163143] up:boot 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:27:17.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:17 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:27:17.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: pgmap v521: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 12 op/s 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6834/3612423816,v1:172.21.15.193:6837/3612423816] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6835/3660781575,v1:172.21.15.193:6838/3660781575] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.193:6836/946028959,v1:172.21.15.193:6839/946028959] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6834/4183609759,v1:172.21.15.148:6837/4183609759] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6835/1595305815,v1:172.21.15.148:6838/1595305815] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: mds.? [v2:172.21.15.148:6836/3334163143,v1:172.21.15.148:6839/3334163143] up:boot 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='mgr.15225 172.21.15.78:0/894179351' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-16T01:27:17.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:17 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-16T01:27:18.010 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:27:18.256 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:18.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:18 smithi078 ceph-mon[30937]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T01:27:18.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:18 smithi078 ceph-mon[30937]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T01:27:18.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:18 smithi078 ceph-mon[30937]: Cluster is now healthy 2024-09-16T01:27:18.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:18 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:27:18.593 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:18 smithi078 ceph-mon[30937]: fsmap 12 up:standby 2024-09-16T01:27:18.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:18 smithi148 ceph-mon[35030]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T01:27:18.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:18 smithi148 ceph-mon[35030]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T01:27:18.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:18 smithi148 ceph-mon[35030]: Cluster is now healthy 2024-09-16T01:27:18.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:18 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:27:18.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:18 smithi148 ceph-mon[35030]: fsmap 12 up:standby 2024-09-16T01:27:18.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:18 smithi193 ceph-mon[35701]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-16T01:27:18.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:18 smithi193 ceph-mon[35701]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-16T01:27:18.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:18 smithi193 ceph-mon[35701]: Cluster is now healthy 2024-09-16T01:27:18.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:18 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/87373892' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-16T01:27:18.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:18 smithi193 ceph-mon[35701]: fsmap 12 up:standby 2024-09-16T01:27:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 -- 172.21.15.78:0/3054016446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e810e4f0 msgr2=0x7f08e8110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 --2- 172.21.15.78:0/3054016446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e810e4f0 0x7f08e8110900 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009ca0 tx=0x7f08e002a8c0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 -- 172.21.15.78:0/3054016446 shutdown_connections 2024-09-16T01:27:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 --2- 172.21.15.78:0/3054016446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e810e4f0 0x7f08e8110900 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 --2- 172.21.15.78:0/3054016446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e8101380 0x7f08e810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 --2- 172.21.15.78:0/3054016446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e8100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 -- 172.21.15.78:0/3054016446 >> 172.21.15.78:0/3054016446 conn(0x7f08e80fc7d0 msgr2=0x7f08e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 -- 172.21.15.78:0/3054016446 shutdown_connections 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.918+0000 7f08f02b4640 1 -- 172.21.15.78:0/3054016446 wait complete. 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 Processor -- start 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 -- start start 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e81a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e810e4f0 0x7f08e819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81142d0 con 0x7f08e8101380 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f08e8114150 con 0x7f08e81009d0 2024-09-16T01:27:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08f02b4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08e8114450 con 0x7f08e810e4f0 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08ed84f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08ed84f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33634/0 (socket says 172.21.15.78:33634) 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08ed84f640 1 -- 172.21.15.78:0/193887252 learned_addr learned my addr 172.21.15.78:0/193887252 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08ee851640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e810e4f0 0x7f08e819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.919+0000 7f08ee050640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e81a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 -- 172.21.15.78:0/193887252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e810e4f0 msgr2=0x7f08e819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e810e4f0 0x7f08e819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 -- 172.21.15.78:0/193887252 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 msgr2=0x7f08e81a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e81a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 -- 172.21.15.78:0/193887252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e819ad20 con 0x7f08e8101380 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ee050640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e81a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08ed84f640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f08d800d770 tx=0x7f08d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8017070 con 0x7f08e8101380 2024-09-16T01:27:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e819b010 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08d8005500 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8004e00 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.920+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e81a7300 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.921+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e806a7d0 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.922+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f08d8004fa0 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.923+0000 7f08df7fe640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 0x7f08d40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.923+0000 7f08ee050640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 0x7f08d40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.923+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9695+0+0 (secure 0 0 0) 0x7f08d80cfc30 con 0x7f08e8101380 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.923+0000 7f08ee050640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 0x7f08d40a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009930 tx=0x7f08e4008040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:18.927+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f08d809c180 con 0x7f08e8101380 2024-09-16T01:27:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.097+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f08e8109680 con 0x7f08e8101380 2024-09-16T01:27:19.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.315+0000 7f08df7fe640 1 -- 172.21.15.78:0/193887252 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) ==== 181+0+0 (secure 0 0 0) 0x7f08d809ba80 con 0x7f08e8101380 2024-09-16T01:27:19.317 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 msgr2=0x7f08d40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 0x7f08d40a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009930 tx=0x7f08e4008040 comp rx=0 tx=0).stop 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 msgr2=0x7f08e81a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f08d800d770 tx=0x7f08d800dc30 comp rx=0 tx=0).stop 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 shutdown_connections 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f08d40a33b0 0x7f08d40a5870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08e810e4f0 0x7f08e819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08e8101380 0x7f08e81a0970 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 --2- 172.21.15.78:0/193887252 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08e81009d0 0x7f08e81a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.318+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 >> 172.21.15.78:0/193887252 conn(0x7f08e80fc7d0 msgr2=0x7f08e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.319+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 shutdown_connections 2024-09-16T01:27:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:19.319+0000 7f08f02b4640 1 -- 172.21.15.78:0/193887252 wait complete. 2024-09-16T01:27:19.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:19 smithi148 ceph-mon[35030]: pgmap v522: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 17 KiB/s wr, 9 op/s 2024-09-16T01:27:19.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:19 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:19.699 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:19 smithi078 ceph-mon[30937]: pgmap v522: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 17 KiB/s wr, 9 op/s 2024-09-16T01:27:19.700 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:19 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:19.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:19 smithi193 ceph-mon[35701]: pgmap v522: 129 pgs: 129 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 17 KiB/s wr, 9 op/s 2024-09-16T01:27:19.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:19 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:20.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:27:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:20.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:20 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:20.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:20 smithi078 ceph-mon[30937]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T01:27:20.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:20 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:20.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:20 smithi148 ceph-mon[35030]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T01:27:20.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:20 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/193887252' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:20.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:20 smithi193 ceph-mon[35701]: osdmap e119: 12 total, 12 up, 12 in 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 -- 172.21.15.78:0/905471793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4004103660 msgr2=0x7f400410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 --2- 172.21.15.78:0/905471793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4004103660 0x7f400410bb50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009900 tx=0x7f3ff402a480 comp rx=0 tx=0).stop 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 -- 172.21.15.78:0/905471793 shutdown_connections 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 --2- 172.21.15.78:0/905471793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f400410c170 0x7f400410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 --2- 172.21.15.78:0/905471793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4004103660 0x7f400410bb50 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 --2- 172.21.15.78:0/905471793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004102d40 0x7f4004103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 -- 172.21.15.78:0/905471793 >> 172.21.15.78:0/905471793 conn(0x7f40040fc7a0 msgr2=0x7f40040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 -- 172.21.15.78:0/905471793 shutdown_connections 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.802+0000 7f4008fa7640 1 -- 172.21.15.78:0/905471793 wait complete. 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.803+0000 7f4008fa7640 1 Processor -- start 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.803+0000 7f4008fa7640 1 -- start start 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 0x7f4004198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004103660 0x7f4004198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4004112030 con 0x7f400410c170 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4004111eb0 con 0x7f4004103660 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40041121b0 con 0x7f4004102d40 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33640/0 (socket says 172.21.15.78:33640) 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 -- 172.21.15.78:0/236766139 learned_addr learned my addr 172.21.15.78:0/236766139 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4001d9b640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004103660 0x7f4004198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f400259c640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 0x7f4004198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 -- 172.21.15.78:0/236766139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 msgr2=0x7f4004198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 0x7f4004198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 -- 172.21.15.78:0/236766139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004103660 msgr2=0x7f4004198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004103660 0x7f4004198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 -- 172.21.15.78:0/236766139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400419d530 con 0x7f400410c170 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f400259c640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 0x7f4004198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4002d9d640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00b970 tx=0x7f3ffc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc019070 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc00cb40 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400419d820 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc007650 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.804+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40041a5100 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.805+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40041031a0 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.806+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f3ffc01c070 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.807+0000 7f3ffb7fe640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 0x7f3ff00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.807+0000 7f400259c640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 0x7f3ff00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.807+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9170+0+0 (secure 0 0 0) 0x7f3ffc0cf5c0 con 0x7f400410c170 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.807+0000 7f400259c640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 0x7f3ff00a5870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3fec006fd0 tx=0x7f3fec008040 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.811+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ffc09bdc0 con 0x7f400410c170 2024-09-16T01:27:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:20.978+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f400402d7c0 con 0x7f400410c170 2024-09-16T01:27:21.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:21 smithi148 ceph-mon[35030]: pgmap v524: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:21.636 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:21 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:21.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.643+0000 7f3ffb7fe640 1 -- 172.21.15.78:0/236766139 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) ==== 169+0+0 (secure 0 0 0) 0x7f3ffc09b6c0 con 0x7f400410c170 2024-09-16T01:27:21.646 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-09-16T01:27:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:21 smithi078 ceph-mon[30937]: pgmap v524: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:21.646 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:21 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 msgr2=0x7f3ff00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 0x7f3ff00a5870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3fec006fd0 tx=0x7f3fec008040 comp rx=0 tx=0).stop 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 msgr2=0x7f400419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00b970 tx=0x7f3ffc00be30 comp rx=0 tx=0).stop 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 shutdown_connections 2024-09-16T01:27:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f3ff00a33b0 0x7f3ff00a5870 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f400410c170 0x7f400419ce50 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4004103660 0x7f4004198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 --2- 172.21.15.78:0/236766139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4004102d40 0x7f4004198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 >> 172.21.15.78:0/236766139 conn(0x7f40040fc7a0 msgr2=0x7f40040fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 shutdown_connections 2024-09-16T01:27:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:21.646+0000 7f4008fa7640 1 -- 172.21.15.78:0/236766139 wait complete. 2024-09-16T01:27:21.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:21 smithi193 ceph-mon[35701]: pgmap v524: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:21.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:21 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:22.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid fa4731de-73c6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-16T01:27:22.656 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/mon.a/config 2024-09-16T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:22 smithi078 ceph-mon[30937]: pgmap v525: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:22 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:22.937 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:22 smithi078 ceph-mon[30937]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T01:27:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:22 smithi193 ceph-mon[35701]: pgmap v525: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:22 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:23.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:22 smithi193 ceph-mon[35701]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T01:27:23.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:22 smithi148 ceph-mon[35030]: pgmap v525: 97 pgs: 97 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:23.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:22 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/236766139' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:23.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:22 smithi148 ceph-mon[35030]: osdmap e120: 12 total, 12 up, 12 in 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 -- 172.21.15.78:0/3166493585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4100350 msgr2=0x7f00a4100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/3166493585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4100350 0x7f00a4100730 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0094009cd0 tx=0x7f009402a8e0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 -- 172.21.15.78:0/3166493585 shutdown_connections 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/3166493585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4106f50 0x7f00a410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/3166493585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 0x7f00a4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/3166493585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4100350 0x7f00a4100730 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 -- 172.21.15.78:0/3166493585 >> 172.21.15.78:0/3166493585 conn(0x7f00a4077210 msgr2=0x7f00a40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 -- 172.21.15.78:0/3166493585 shutdown_connections 2024-09-16T01:27:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.152+0000 7f00a9c9b640 1 -- 172.21.15.78:0/3166493585 wait complete. 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 Processor -- start 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 -- start start 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4100350 0x7f00a41012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 0x7f00a4101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4111fa0 con 0x7f00a4106f50 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4111e20 con 0x7f00a4100350 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a9c9b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4112120 con 0x7f00a4100d00 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.153+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33658/0 (socket says 172.21.15.78:33658) 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 -- 172.21.15.78:0/2410861760 learned_addr learned my addr 172.21.15.78:0/2410861760 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-09-16T01:27:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a37fe640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4100350 0x7f00a41012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a2ffd640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 0x7f00a4101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 -- 172.21.15.78:0/2410861760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 msgr2=0x7f00a4101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 0x7f00a4101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 -- 172.21.15.78:0/2410861760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4100350 msgr2=0x7f00a41012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4100350 0x7f00a41012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 -- 172.21.15.78:0/2410861760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a41a6a20 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a3fff640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f009c00d950 tx=0x7f009c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c017030 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009c004020 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a41a6d10 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c003c00 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.154+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a41a7250 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.155+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a4100730 con 0x7f00a4106f50 2024-09-16T01:27:23.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.156+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) ==== 138486+0+0 (secure 0 0 0) 0x7f009c01a070 con 0x7f00a4106f50 2024-09-16T01:27:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.157+0000 7f00a0ff9640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 0x7f008c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:27:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.157+0000 7f00a37fe640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 0x7f008c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:27:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.158+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8582+0+0 (secure 0 0 0) 0x7f009c0cf820 con 0x7f00a4106f50 2024-09-16T01:27:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.158+0000 7f00a37fe640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 0x7f008c0a58f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0094006010 tx=0x7f0094009360 comp rx=0 tx=0).ready entity=mgr.15225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:27:23.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.161+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f009c021070 con 0x7f00a4106f50 2024-09-16T01:27:23.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.330+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f00a4102800 con 0x7f00a4106f50 2024-09-16T01:27:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.649+0000 7f00a0ff9640 1 -- 172.21.15.78:0/2410861760 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v121) ==== 178+0+0 (secure 0 0 0) 0x7f00a4102800 con 0x7f00a4106f50 2024-09-16T01:27:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2024-09-16T01:27:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 msgr2=0x7f008c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 0x7f008c0a58f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0094006010 tx=0x7f0094009360 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 msgr2=0x7f00a4101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f009c00d950 tx=0x7f009c00de10 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 shutdown_connections 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:6800/2602873484,v1:172.21.15.78:6801/2602873484] conn(0x7f008c0a3430 0x7f008c0a58f0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0094006010 tx=0x7f0094009360 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00a4106f50 0x7f00a4101d50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00a4100d00 0x7f00a4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 --2- 172.21.15.78:0/2410861760 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00a4100350 0x7f00a41012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:27:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.653+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 >> 172.21.15.78:0/2410861760 conn(0x7f00a4077210 msgr2=0x7f00a41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:27:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.654+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 shutdown_connections 2024-09-16T01:27:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-16T01:27:23.654+0000 7f00a9c9b640 1 -- 172.21.15.78:0/2410861760 wait complete. 2024-09-16T01:27:24.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:23 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:24.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:23 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:24.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:23 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-16T01:27:25.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:24 smithi078 ceph-mon[30937]: pgmap v527: 33 pgs: 33 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:24 smithi078 ceph-mon[30937]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:25.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:24 smithi078 ceph-mon[30937]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T01:27:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:24 smithi193 ceph-mon[35701]: pgmap v527: 33 pgs: 33 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:24 smithi193 ceph-mon[35701]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:25.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:24 smithi193 ceph-mon[35701]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T01:27:25.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:24 smithi148 ceph-mon[35030]: pgmap v527: 33 pgs: 33 active+clean; 2.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:25.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:24 smithi148 ceph-mon[35030]: from='client.? 172.21.15.78:0/2410861760' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-16T01:27:25.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:24 smithi148 ceph-mon[35030]: osdmap e121: 12 total, 12 up, 12 in 2024-09-16T01:27:27.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:26 smithi078 ceph-mon[30937]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:27.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:26 smithi193 ceph-mon[35701]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:27.135 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:26 smithi148 ceph-mon[35030]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:27.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:27:27.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:27:27.474 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:27:27.520 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-16T01:27:27.521 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T01:27:27.649 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T01:27:27.666 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-16T01:27:27.682 INFO:tasks.cephadm:Stopping all daemons... 2024-09-16T01:27:27.683 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-16T01:27:27.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a 2024-09-16T01:27:28.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:27 smithi078 systemd[1]: Stopping Ceph mon.a for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:29.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:28 smithi078 ceph-mon[30937]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:29.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:28 smithi193 ceph-mon[35701]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:29.136 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:28 smithi148 ceph-mon[35030]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-16T01:27:30.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a[30933]: 2024-09-16T01:27:29.620+0000 7f40b87d4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:30.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a[30933]: 2024-09-16T01:27:29.620+0000 7f40b87d4640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-16T01:27:30.029 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:29 smithi078 podman[104475]: 2024-09-16 01:27:29.641628645 +0000 UTC m=+1.569045450 container died edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:27:35.242 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:34 smithi078 podman[104475]: 2024-09-16 01:27:34.779553788 +0000 UTC m=+6.706970625 container cleanup edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:27:35.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:34 smithi078 bash[104475]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a 2024-09-16T01:27:35.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Sep 16 01:27:35 smithi078 podman[104486]: 2024-09-16 01:27:35.023059881 +0000 UTC m=+5.377020954 container remove edc50c2d55b710c796986f9e56344d6070e77d5f0c609358295ba37c7cb1e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:27:35.264 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.a.service' 2024-09-16T01:27:35.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:35.319 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-16T01:27:35.319 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-16T01:27:35.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.b 2024-09-16T01:27:35.635 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:35 smithi148 systemd[1]: Stopping Ceph mon.b for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:36.081 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-b[35026]: 2024-09-16T01:27:35.706+0000 7fc0f59bf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:36.081 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-b[35026]: 2024-09-16T01:27:35.706+0000 7fc0f59bf640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-16T01:27:36.081 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:35 smithi148 podman[71175]: 2024-09-16 01:27:35.789533158 +0000 UTC m=+0.221423896 container died 517650b498435b331d52152996d1422520d62f82d6b623deed8e50e6d32a105f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-b, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-16T01:27:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:36 smithi148 podman[71175]: 2024-09-16 01:27:36.175267183 +0000 UTC m=+0.607157928 container cleanup 517650b498435b331d52152996d1422520d62f82d6b623deed8e50e6d32a105f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-16T01:27:36.386 INFO:journalctl@ceph.mon.b.smithi148.stdout:Sep 16 01:27:36 smithi148 bash[71175]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-b 2024-09-16T01:27:36.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.b.service' 2024-09-16T01:27:36.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:36.638 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-16T01:27:36.638 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-16T01:27:36.638 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.c 2024-09-16T01:27:37.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:36 smithi193 systemd[1]: Stopping Ceph mon.c for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:37.296 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c[35697]: 2024-09-16T01:27:37.012+0000 7fd4eef17640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:37.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c[35697]: 2024-09-16T01:27:37.012+0000 7fd4eef17640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-16T01:27:37.297 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:37 smithi193 podman[73225]: 2024-09-16 01:27:37.248245438 +0000 UTC m=+0.373161434 container died 18b55d0005518ac31a12b8c7819356d4b87c934ac90b1d4a632e90802d4139b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-16T01:27:38.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:37 smithi193 podman[73225]: 2024-09-16 01:27:37.88194013 +0000 UTC m=+1.006856080 container cleanup 18b55d0005518ac31a12b8c7819356d4b87c934ac90b1d4a632e90802d4139b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:27:38.046 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:37 smithi193 bash[73225]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c 2024-09-16T01:27:38.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.c.service' 2024-09-16T01:27:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:38 smithi193 podman[73240]: 2024-09-16 01:27:38.242122281 +0000 UTC m=+0.986480201 container remove 18b55d0005518ac31a12b8c7819356d4b87c934ac90b1d4a632e90802d4139b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:27:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:38 smithi193 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-16T01:27:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:38 smithi193 systemd[1]: Stopped Ceph mon.c for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:27:38.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Sep 16 01:27:38 smithi193 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mon.c.service: Consumed 14.368s CPU time. 2024-09-16T01:27:38.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:38.534 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-16T01:27:38.535 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-16T01:27:38.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x 2024-09-16T01:27:38.869 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:27:38 smithi078 systemd[1]: Stopping Ceph mgr.x for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:39.196 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:27:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x[31166]: 2024-09-16T01:27:38.867+0000 7fdb09167640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:39.196 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:27:38 smithi078 podman[104600]: 2024-09-16 01:27:38.898717412 +0000 UTC m=+0.175586281 container died 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-16T01:27:39.528 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:27:39 smithi078 podman[104600]: 2024-09-16 01:27:39.310363785 +0000 UTC m=+0.587232654 container cleanup 12677b9a0f03648da3fcd42745cf67facc7c6b511b4240eb632823727823c972 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-16T01:27:39.529 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Sep 16 01:27:39 smithi078 bash[104600]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-x 2024-09-16T01:27:39.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-16T01:27:39.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:39.926 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-16T01:27:39.926 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-16T01:27:39.927 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.y 2024-09-16T01:27:40.270 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:27:39 smithi148 systemd[1]: Stopping Ceph mgr.y for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:40.557 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:27:40 smithi148 podman[71299]: 2024-09-16 01:27:40.269181332 +0000 UTC m=+0.149955100 container died 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True) 2024-09-16T01:27:40.886 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:27:40 smithi148 podman[71299]: 2024-09-16 01:27:40.656294818 +0000 UTC m=+0.537068580 container cleanup 639eed3ea3448e806fc6a88f8e1faedf2c9e28bd7dda71651c0d5cfed64a10a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-16T01:27:40.886 INFO:journalctl@ceph.mgr.y.smithi148.stdout:Sep 16 01:27:40 smithi148 bash[71299]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-y 2024-09-16T01:27:41.232 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-16T01:27:41.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:41.269 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-16T01:27:41.270 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-16T01:27:41.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.z 2024-09-16T01:27:41.744 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:27:41 smithi193 systemd[1]: Stopping Ceph mgr.z for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:42.039 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:27:41 smithi193 podman[73360]: 2024-09-16 01:27:41.742972248 +0000 UTC m=+0.263229233 container died 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:27:42.296 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:27:42 smithi193 podman[73360]: 2024-09-16 01:27:42.112456078 +0000 UTC m=+0.632713060 container cleanup 1822b141934b0e235d530f8b528e16b4e1d7b88c8b8e5d12887aec9b28ab98a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-16T01:27:42.296 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Sep 16 01:27:42 smithi193 bash[73360]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mgr-z 2024-09-16T01:27:42.536 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-16T01:27:42.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:42.573 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-16T01:27:42.573 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-16T01:27:42.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.0 2024-09-16T01:27:42.891 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:42 smithi078 systemd[1]: Stopping Ceph osd.0 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:43.278 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:27:42.889+0000 7f6470ce1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:43.279 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:27:42.889+0000 7f6470ce1640 -1 osd.0 121 *** Got signal Terminated *** 2024-09-16T01:27:43.279 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0[45198]: 2024-09-16T01:27:42.890+0000 7f6470ce1640 -1 osd.0 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:27:48.779 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:48 smithi078 podman[104724]: 2024-09-16 01:27:48.518057517 +0000 UTC m=+5.746055580 container died fffbae5ba777cb93f0879cea2878401de9bbda2b745aa9e6c71662e33276c7e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-16T01:27:49.219 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:48 smithi078 podman[104724]: 2024-09-16 01:27:48.908023959 +0000 UTC m=+6.136022020 container cleanup fffbae5ba777cb93f0879cea2878401de9bbda2b745aa9e6c71662e33276c7e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-16T01:27:49.219 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:48 smithi078 bash[104724]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0 2024-09-16T01:27:49.528 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104749]: 2024-09-16 01:27:49.218173436 +0000 UTC m=+0.699111214 container remove fffbae5ba777cb93f0879cea2878401de9bbda2b745aa9e6c71662e33276c7e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-16T01:27:49.855 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104843]: 2024-09-16 01:27:49.431504361 +0000 UTC m=+0.015727989 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:27:49.856 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104843]: 2024-09-16 01:27:49.560759184 +0000 UTC m=+0.144982803 container create cb44d13819ed2c7d95f1237b44356d6423ad675390a47ea3c22616820eb73795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:27:50.250 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104843]: 2024-09-16 01:27:49.855172286 +0000 UTC m=+0.439395904 container init cb44d13819ed2c7d95f1237b44356d6423ad675390a47ea3c22616820eb73795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-16T01:27:50.251 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104843]: 2024-09-16 01:27:49.858615993 +0000 UTC m=+0.442839612 container start cb44d13819ed2c7d95f1237b44356d6423ad675390a47ea3c22616820eb73795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:27:50.251 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:49 smithi078 podman[104843]: 2024-09-16 01:27:49.980685772 +0000 UTC m=+0.564909403 container attach cb44d13819ed2c7d95f1237b44356d6423ad675390a47ea3c22616820eb73795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-16T01:27:50.528 INFO:journalctl@ceph.osd.0.smithi078.stdout:Sep 16 01:27:50 smithi078 podman[104843]: 2024-09-16 01:27:50.251039483 +0000 UTC m=+0.835263120 container died cb44d13819ed2c7d95f1237b44356d6423ad675390a47ea3c22616820eb73795 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-16T01:27:50.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.0.service' 2024-09-16T01:27:50.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:50.977 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-16T01:27:50.977 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-16T01:27:50.977 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.1 2024-09-16T01:27:51.361 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:51 smithi148 systemd[1]: Stopping Ceph osd.1 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:27:51.635 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:27:51.359+0000 7f8159ab7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:27:51.636 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:27:51.359+0000 7f8159ab7640 -1 osd.1 121 *** Got signal Terminated *** 2024-09-16T01:27:51.636 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1[39462]: 2024-09-16T01:27:51.359+0000 7f8159ab7640 -1 osd.1 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:27:57.437 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:57 smithi148 podman[71422]: 2024-09-16 01:27:57.143390057 +0000 UTC m=+5.976715734 container died dea72cdb0df788cdb8f186193d33579b5fbd113bb09012626f98dbfd5d1180ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-16T01:27:57.804 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:57 smithi148 podman[71422]: 2024-09-16 01:27:57.526763204 +0000 UTC m=+6.360088889 container cleanup dea72cdb0df788cdb8f186193d33579b5fbd113bb09012626f98dbfd5d1180ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T01:27:57.804 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:57 smithi148 bash[71422]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1 2024-09-16T01:27:58.136 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:57 smithi148 podman[71437]: 2024-09-16 01:27:57.803456559 +0000 UTC m=+0.660240958 container remove dea72cdb0df788cdb8f186193d33579b5fbd113bb09012626f98dbfd5d1180ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T01:27:58.441 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.060977709 +0000 UTC m=+0.016311673 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:27:58.441 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.187715346 +0000 UTC m=+0.143049355 container create 3fc452bcd8f083ca3fe01d09bfd169089a8cf7ca01405525ce411cc99c2dfaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-16T01:27:58.441 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.440625058 +0000 UTC m=+0.395959023 container init 3fc452bcd8f083ca3fe01d09bfd169089a8cf7ca01405525ce411cc99c2dfaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:27:58.747 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.444308109 +0000 UTC m=+0.399642085 container start 3fc452bcd8f083ca3fe01d09bfd169089a8cf7ca01405525ce411cc99c2dfaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:27:58.747 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.549227535 +0000 UTC m=+0.504561514 container attach 3fc452bcd8f083ca3fe01d09bfd169089a8cf7ca01405525ce411cc99c2dfaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-16T01:27:59.111 INFO:journalctl@ceph.osd.1.smithi148.stdout:Sep 16 01:27:58 smithi148 podman[71523]: 2024-09-16 01:27:58.746362965 +0000 UTC m=+0.701696938 container died 3fc452bcd8f083ca3fe01d09bfd169089a8cf7ca01405525ce411cc99c2dfaed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-16T01:27:59.457 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.1.service' 2024-09-16T01:27:59.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:27:59.499 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-16T01:27:59.500 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-16T01:27:59.500 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.2 2024-09-16T01:27:59.796 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:27:59 smithi193 systemd[1]: Stopping Ceph osd.2 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:00.296 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:27:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:27:59.898+0000 7f4707ac8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:00.296 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:27:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:27:59.898+0000 7f4707ac8640 -1 osd.2 121 *** Got signal Terminated *** 2024-09-16T01:28:00.297 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:27:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2[40653]: 2024-09-16T01:27:59.898+0000 7f4707ac8640 -1 osd.2 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:05.961 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:05 smithi193 podman[73483]: 2024-09-16 01:28:05.653942114 +0000 UTC m=+5.954713328 container died 9b5c4e3651a894c74a12219b0927cbdfa704d5487d9f19ad0f6a442f30ab98b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T01:28:06.297 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73483]: 2024-09-16 01:28:06.024950941 +0000 UTC m=+6.325722154 container cleanup 9b5c4e3651a894c74a12219b0927cbdfa704d5487d9f19ad0f6a442f30ab98b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-16T01:28:06.297 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 bash[73483]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2 2024-09-16T01:28:06.639 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73498]: 2024-09-16 01:28:06.30156429 +0000 UTC m=+0.644534488 container remove 9b5c4e3651a894c74a12219b0927cbdfa704d5487d9f19ad0f6a442f30ab98b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-16T01:28:06.914 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73588]: 2024-09-16 01:28:06.541365347 +0000 UTC m=+0.016036175 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:06.914 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73588]: 2024-09-16 01:28:06.660954086 +0000 UTC m=+0.135624914 container create ea6087e2ac7e862363b24958de1e7673a4ff96b43223b2fbf9fa143d1b5ec4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-16T01:28:06.915 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73588]: 2024-09-16 01:28:06.913669776 +0000 UTC m=+0.388340603 container init ea6087e2ac7e862363b24958de1e7673a4ff96b43223b2fbf9fa143d1b5ec4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:28:07.184 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:06 smithi193 podman[73588]: 2024-09-16 01:28:06.917577938 +0000 UTC m=+0.392248775 container start ea6087e2ac7e862363b24958de1e7673a4ff96b43223b2fbf9fa143d1b5ec4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T01:28:07.184 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:07 smithi193 podman[73588]: 2024-09-16 01:28:07.022119998 +0000 UTC m=+0.496790839 container attach ea6087e2ac7e862363b24958de1e7673a4ff96b43223b2fbf9fa143d1b5ec4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-16T01:28:07.184 INFO:journalctl@ceph.osd.2.smithi193.stdout:Sep 16 01:28:07 smithi193 podman[73588]: 2024-09-16 01:28:07.160764017 +0000 UTC m=+0.635434852 container died ea6087e2ac7e862363b24958de1e7673a4ff96b43223b2fbf9fa143d1b5ec4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:28:07.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.2.service' 2024-09-16T01:28:07.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:07.799 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-16T01:28:07.799 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-16T01:28:07.800 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.3 2024-09-16T01:28:08.139 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:07 smithi078 systemd[1]: Stopping Ceph osd.3 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:08.528 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:08.137+0000 7f43d6343640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:08.528 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:08.137+0000 7f43d6343640 -1 osd.3 121 *** Got signal Terminated *** 2024-09-16T01:28:08.529 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:08.137+0000 7f43d6343640 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:09.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:08 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:08.882+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:10.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:09 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:09.879+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:10.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:10.075+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:11.028 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:10 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:10.720+0000 7f43ce951640 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:45.045343+0000 front 2024-09-16T01:27:45.045320+0000 (oldest deadline 2024-09-16T01:28:09.745458+0000) 2024-09-16T01:28:11.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:10 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:10.854+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:11.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:11.220+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:11.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:11.071+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:11.599 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:11 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:11.195+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:12.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:11.892+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:12.029 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:11 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:11.706+0000 7f43ce951640 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:45.045343+0000 front 2024-09-16T01:27:45.045320+0000 (oldest deadline 2024-09-16T01:28:09.745458+0000) 2024-09-16T01:28:12.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:12.025+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:12.301 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:12.193+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:12.635 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:12 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:12.191+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:13.028 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3[52102]: 2024-09-16T01:28:12.727+0000 7f43ce951640 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:45.045343+0000 front 2024-09-16T01:27:45.045320+0000 (oldest deadline 2024-09-16T01:28:09.745458+0000) 2024-09-16T01:28:13.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:12 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:12.877+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:13.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:13 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:13.169+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:13.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:13.056+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:13.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:13 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:13.237+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:14.175 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:13 smithi078 podman[104945]: 2024-09-16 01:28:13.998093051 +0000 UTC m=+6.006151274 container died 16f9c7cfcf2177a5107fb8789fca76c68753c676f418d3dd1432d38b069cc723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:28:14.175 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:13 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:13.921+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:14.429 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:14 smithi078 podman[104945]: 2024-09-16 01:28:14.413550347 +0000 UTC m=+6.421608569 container cleanup 16f9c7cfcf2177a5107fb8789fca76c68753c676f418d3dd1432d38b069cc723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=) 2024-09-16T01:28:14.429 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:14 smithi078 bash[104945]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3 2024-09-16T01:28:14.430 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:14.174+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:14.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:14 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:14.080+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:14.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:14 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:14.258+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:14.936 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:14 smithi078 podman[104971]: 2024-09-16 01:28:14.681754413 +0000 UTC m=+0.686808026 container remove 16f9c7cfcf2177a5107fb8789fca76c68753c676f418d3dd1432d38b069cc723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:15.213 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.077061449 +0000 UTC m=+0.016065625 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:15.213 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:14 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:14.934+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:15.485 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.213013138 +0000 UTC m=+0.152017312 container create 75cfba3eedafb218f1474406a2e278ba4fc33f37978486dba7b1c58d039991a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T01:28:15.486 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:15 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:15.212+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:15.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:15.096+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:15.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:15 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:15.257+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:15.779 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.485347846 +0000 UTC m=+0.424352015 container init 75cfba3eedafb218f1474406a2e278ba4fc33f37978486dba7b1c58d039991a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-16T01:28:15.779 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.488675506 +0000 UTC m=+0.427679675 container start 75cfba3eedafb218f1474406a2e278ba4fc33f37978486dba7b1c58d039991a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-16T01:28:15.779 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.585458815 +0000 UTC m=+0.524462998 container attach 75cfba3eedafb218f1474406a2e278ba4fc33f37978486dba7b1c58d039991a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-16T01:28:15.779 INFO:journalctl@ceph.osd.3.smithi078.stdout:Sep 16 01:28:15 smithi078 podman[105063]: 2024-09-16 01:28:15.678145073 +0000 UTC m=+0.617149265 container died 75cfba3eedafb218f1474406a2e278ba4fc33f37978486dba7b1c58d039991a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:28:16.219 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:15 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:15.931+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:16.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.3.service' 2024-09-16T01:28:16.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:16.412 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-16T01:28:16.412 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-16T01:28:16.412 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.4 2024-09-16T01:28:16.467 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:16 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:16.210+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:16.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:16 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:16.218+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:16.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:16.137+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:16.865 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:16 smithi148 systemd[1]: Stopping Ceph osd.4 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:17.135 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:16 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:16.863+0000 7fe4e05ef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:17.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:16 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:16.863+0000 7fe4e05ef640 -1 osd.4 121 *** Got signal Terminated *** 2024-09-16T01:28:17.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:16 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:16.863+0000 7fe4e05ef640 -1 osd.4 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:17.179 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:16 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:16.900+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:17.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:17 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:17.178+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:17.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:17.119+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:17.635 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:17 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:17.184+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:18.046 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:17.720+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:18.227 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:17 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:17.889+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:18.227 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:17 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:17.889+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:18.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:18.226+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:18.529 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:18 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:18.228+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:18.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:18.077+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:18.885 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:18 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:18.585+0000 7fe4d8bfd640 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:53.261475+0000 front 2024-09-16T01:27:53.261569+0000 (oldest deadline 2024-09-16T01:28:18.462180+0000) 2024-09-16T01:28:18.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:18 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:18.528+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:19.046 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:18.732+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:19.260 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:18.937+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:19.260 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:18 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:18.937+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:19.519 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:19 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:19.193+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:19.519 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:19 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:19.193+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:19.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:19 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:19.258+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:19.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:19 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:19.078+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:19.886 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:19 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:19.589+0000 7fe4d8bfd640 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:53.261475+0000 front 2024-09-16T01:27:53.261569+0000 (oldest deadline 2024-09-16T01:28:18.462180+0000) 2024-09-16T01:28:19.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:19 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:19.517+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:20.046 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:19 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:19.739+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:20.212 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:19 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:19.903+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:20.212 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:19 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:19.903+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:20.507 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:20 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:20.189+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:20.508 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:20 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:20.189+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:20.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:20 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:20.211+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:20.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:20 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:20.211+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:20.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:20 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:20.060+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:20.886 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:20 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:20.550+0000 7fe4d8bfd640 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:53.261475+0000 front 2024-09-16T01:27:53.261569+0000 (oldest deadline 2024-09-16T01:28:18.462180+0000) 2024-09-16T01:28:20.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:20 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:20.505+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:21.046 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:20 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:20.786+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:21.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:20 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:20.900+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:21.202 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:20 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:20.875+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:21.202 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:20 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:20.875+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:21.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:21 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:21.201+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:21.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:21 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:21.201+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:21.533 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:21 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:21.460+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:21.533 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:21 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:21.238+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:21.533 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:21 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:21.238+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:21.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:21 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:21.094+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:21.886 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:21 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4[45932]: 2024-09-16T01:28:21.531+0000 7fe4d8bfd640 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:53.261475+0000 front 2024-09-16T01:27:53.261569+0000 (oldest deadline 2024-09-16T01:28:18.462180+0000) 2024-09-16T01:28:22.082 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:21 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:21.814+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:22.082 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:21 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:21.903+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:22.192 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:21 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:21.830+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:22.192 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:21 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:21.830+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:22.498 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:22 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:22.194+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:22.498 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:22 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:22.194+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:22.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:22.190+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:22.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:22.190+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:22.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:22 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:22.081+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:22.886 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:22 smithi148 podman[71624]: 2024-09-16 01:28:22.652317563 +0000 UTC m=+6.038161404 container died ec6fd1dafcb05334d80336b14883b31ca59b049baa5d639733ad7e4fd13655d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-16T01:28:22.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:22 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:22.496+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:23.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:22.797+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:23.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:22.797+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:23.114 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:22 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:22.838+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:23.115 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:22 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:22.889+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:23.308 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71624]: 2024-09-16 01:28:23.055911097 +0000 UTC m=+6.441754941 container cleanup ec6fd1dafcb05334d80336b14883b31ca59b049baa5d639733ad7e4fd13655d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:28:23.308 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 bash[71624]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4 2024-09-16T01:28:23.308 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:23 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:23.161+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:23.308 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:23 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:23.161+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:23.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:23 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:23.233+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:23.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:23 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:23.233+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:23.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:23 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:23.113+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:23.617 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:23 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:23.507+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:23.617 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71642]: 2024-09-16 01:28:23.307243452 +0000 UTC m=+0.653727599 container remove ec6fd1dafcb05334d80336b14883b31ca59b049baa5d639733ad7e4fd13655d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-16T01:28:23.870 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71728]: 2024-09-16 01:28:23.616595702 +0000 UTC m=+0.111813462 container create d5455f7b72807c28f976ac165460e5ec3091874fe86eb2146a1aec7e4939e7df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-16T01:28:23.870 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71728]: 2024-09-16 01:28:23.520710301 +0000 UTC m=+0.015928036 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:24.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71728]: 2024-09-16 01:28:23.869508164 +0000 UTC m=+0.364725889 container init d5455f7b72807c28f976ac165460e5ec3091874fe86eb2146a1aec7e4939e7df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-09-16T01:28:24.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71728]: 2024-09-16 01:28:23.873163576 +0000 UTC m=+0.368381293 container start d5455f7b72807c28f976ac165460e5ec3091874fe86eb2146a1aec7e4939e7df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:24.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:23 smithi148 podman[71728]: 2024-09-16 01:28:23.96098332 +0000 UTC m=+0.456201041 container attach d5455f7b72807c28f976ac165460e5ec3091874fe86eb2146a1aec7e4939e7df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-16T01:28:24.136 INFO:journalctl@ceph.osd.4.smithi148.stdout:Sep 16 01:28:24 smithi148 podman[71728]: 2024-09-16 01:28:24.055121791 +0000 UTC m=+0.550339505 container died d5455f7b72807c28f976ac165460e5ec3091874fe86eb2146a1aec7e4939e7df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-16T01:28:24.223 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:23 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:23.797+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:24.223 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:23 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:23.797+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:24.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:23 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:23.924+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:24.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:24 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:24.069+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:24.297 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:23 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:23.874+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:24.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:24.222+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:24.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:24.222+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:24.530 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:24 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:24.179+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:24.530 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:24 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:24.179+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:24.749 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.4.service' 2024-09-16T01:28:24.790 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:24 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:24.528+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:24.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:24.792 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-16T01:28:24.792 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-16T01:28:24.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.5 2024-09-16T01:28:25.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:24.749+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:25.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:24.749+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:25.122 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:25 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:25.058+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:25.122 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:24 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:24.970+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:25.123 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:24 smithi193 systemd[1]: Stopping Ceph osd.5 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:25.123 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:24 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:24.849+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:25.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:25 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:25.236+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:25.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:25 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:25.236+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:25.546 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:25 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:25.120+0000 7f6c8a423640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:25.546 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:25 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:25.120+0000 7f6c8a423640 -1 osd.5 121 *** Got signal Terminated *** 2024-09-16T01:28:25.546 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:25 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:25.120+0000 7f6c8a423640 -1 osd.5 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:25.564 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:25 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:25.178+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:25.564 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:25 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:25.178+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:25.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:25 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:25.563+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:26.264 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:25 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:25.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:26.264 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:25 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:25.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:26.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:25 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:25.896+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:26.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:26 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:26.020+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:26.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:26 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:26.040+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:26.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:26 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:26.262+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:26.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:26 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:26.262+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:26.591 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:26 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:26.212+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:26.591 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:26 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:26.212+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:26.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:26 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:26.589+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:27.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:26 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:26.752+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:27.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:26 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:26.752+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:27.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:26 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:26.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:27.297 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:26 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:26.927+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:27.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:26 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:26.995+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:27.563 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:27 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:27.221+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:27.563 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:27 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:27.221+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:27.564 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:27 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:27.221+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:27.778 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:27 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:27.280+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:27.778 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:27 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:27.280+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:27.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:27 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:27.561+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:28.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:27 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:27.777+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:28.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:27 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:27.777+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:28.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:27 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:27.961+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:28.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:27 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:27.888+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:28.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:27 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:27.998+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:28.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:27 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:27.998+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:28.609 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:28 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:28.253+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:28.609 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:28 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:28.253+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:28.609 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:28 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:28.253+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:28.746 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:28.308+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:28.746 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:28.308+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:28.746 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:28.308+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:28.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:28 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:28.607+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:28.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:28 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:28.607+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:29.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:28.744+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:29.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:28.744+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:29.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:28 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:28.744+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:29.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:28.939+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:29.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:28.939+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:29.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:28.962+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:29.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:28.962+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:29.297 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:28.896+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:29.297 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:28 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:28.896+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.601524+0000 front 2024-09-16T01:28:04.601591+0000 (oldest deadline 2024-09-16T01:28:28.101697+0000) 2024-09-16T01:28:29.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:29.263+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:29.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:29.263+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:29.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:29.263+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:29.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:29 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:29.250+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:29.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:29 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:29.250+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:29.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:29 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:29.250+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:29.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:29 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:29.612+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:29.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:29 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:29.612+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:30.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:29.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:30.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:29.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:30.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:29.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:30.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:29.942+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:51.899078+0000 front 2024-09-16T01:27:51.898824+0000 (oldest deadline 2024-09-16T01:28:17.198953+0000) 2024-09-16T01:28:30.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5[47286]: 2024-09-16T01:28:29.942+0000 7f6c82a31640 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.601524+0000 front 2024-09-16T01:28:04.601591+0000 (oldest deadline 2024-09-16T01:28:28.101697+0000) 2024-09-16T01:28:30.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:29.920+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:30.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:29.920+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:30.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:29.964+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:30.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:29 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:29.964+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:30.622 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:30 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:30.241+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:30.623 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:30 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:30.241+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:30.623 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:30 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:30.241+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:30.712 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:30.289+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:30.712 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:30.289+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:30.712 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:30.289+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:30.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:30 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:30.621+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:30.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:30 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:30.621+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:31.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:30.710+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:31.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:30.710+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:31.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:30 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:30.710+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:31.296 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:31 smithi193 podman[73703]: 2024-09-16 01:28:31.116628818 +0000 UTC m=+6.129942891 container died ed3ca084fa2d4bca83e9ebdd4f23a0d69d097be3fc8dab00587c756fe06f9856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=) 2024-09-16T01:28:31.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:30 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:30.941+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:31.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:30 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:30.941+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:31.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:30 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:30.945+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:31.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:30 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:30.945+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:31.600 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:31 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:31.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:31.600 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:31 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:31.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:31.600 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:31 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:31.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:31.690 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:31.293+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:31.690 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:31.293+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:31.691 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:31.293+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:31.748 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:31 smithi193 podman[73703]: 2024-09-16 01:28:31.496448938 +0000 UTC m=+6.509763011 container cleanup ed3ca084fa2d4bca83e9ebdd4f23a0d69d097be3fc8dab00587c756fe06f9856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-16T01:28:31.748 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:31 smithi193 bash[73703]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5 2024-09-16T01:28:31.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:31 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:31.599+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:31.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:31 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:31.599+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:32.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:31.689+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:32.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:31.689+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:32.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:31 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:31.689+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:32.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:31 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:31.958+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:32.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:31 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:31.958+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:32.046 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:31 smithi193 podman[73718]: 2024-09-16 01:28:31.747793538 +0000 UTC m=+0.631265450 container remove ed3ca084fa2d4bca83e9ebdd4f23a0d69d097be3fc8dab00587c756fe06f9856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:28:32.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:31 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:31.980+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:32.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:31 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:31.980+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:32.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:31.96587548 +0000 UTC m=+0.015844799 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:32.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:32.065172457 +0000 UTC m=+0.115141773 container create 7c89477fd1ba58fa1d95a8d21672a8a15a901c0f301681a6029ee671f8657092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:32.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:32.309503325 +0000 UTC m=+0.359472643 container init 7c89477fd1ba58fa1d95a8d21672a8a15a901c0f301681a6029ee671f8657092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:28:32.314 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:32.313686666 +0000 UTC m=+0.363655984 container start 7c89477fd1ba58fa1d95a8d21672a8a15a901c0f301681a6029ee671f8657092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T01:28:32.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:32.243+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:32.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:32.243+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:32.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:32.243+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:32.570 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:32 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:32.263+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:32.570 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:32 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:32.263+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:32.570 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:32 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:32.263+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:32.796 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:32.409671867 +0000 UTC m=+0.459641186 container attach 7c89477fd1ba58fa1d95a8d21672a8a15a901c0f301681a6029ee671f8657092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-16T01:28:32.797 INFO:journalctl@ceph.osd.5.smithi193.stdout:Sep 16 01:28:32 smithi193 podman[73808]: 2024-09-16 01:28:32.505408658 +0000 UTC m=+0.555377976 container died 7c89477fd1ba58fa1d95a8d21672a8a15a901c0f301681a6029ee671f8657092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD) 2024-09-16T01:28:32.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:32 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:32.569+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:32.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:32 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:32.569+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:33.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:32.647+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:33.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:32.647+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:33.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:32 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:32.647+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:33.091 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:32 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:32.934+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:33.091 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:32 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:32.934+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:33.091 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:32 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:32.959+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:33.092 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:32 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:32.959+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:33.114 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.5.service' 2024-09-16T01:28:33.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:33.153 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-16T01:28:33.153 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-16T01:28:33.154 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.6 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 systemd[1]: Stopping Ceph osd.6 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.274+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.274+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.274+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.466+0000 7f2ac9f82640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.466+0000 7f2ac9f82640 -1 osd.6 121 *** Got signal Terminated *** 2024-09-16T01:28:33.469 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:33.466+0000 7f2ac9f82640 -1 osd.6 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:33.589 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:33.228+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:33.589 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:33.228+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:33.589 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:33.228+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:33.778 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:33.646+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:33.778 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:33.646+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:33.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:33.646+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:33.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:33.588+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:33.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:33 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:33.588+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:34.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:33.997+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:34.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:33.997+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:34.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:33.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:34.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:33 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:33.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:34.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:34.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:34.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:34.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:34.613 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:34.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:34.682 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:34.323+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:34.682 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:34.323+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:34.682 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:34.323+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:34.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:34.612+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:34.885 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:34.612+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:34.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:34 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:34.612+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:35.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:34.679+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:35.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:34.679+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:35.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:34 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:34.679+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:35.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.007+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:35.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.007+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:35.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.007+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:35.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:35.023+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:35.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:35.023+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:35.623 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:35.169+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:35.623 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:35.169+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:35.623 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:35.169+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:35.688 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:35.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:35.688 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:35.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:35.688 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:35.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:35.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:35.622+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:35.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:35.622+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:35.886 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:35 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:35.622+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:36.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:35.686+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:36.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:35.686+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:36.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:35 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:35.686+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:36.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:36.033+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:36.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:36.033+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:36.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.959+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:36.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.959+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:36.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:35 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:35.959+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:36.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:36.211+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:36.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:36.211+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:36.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:36.211+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:36.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:36.211+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:36.713 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:36.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:36.713 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:36.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:36.713 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:36.279+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:37.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:36.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:37.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:36.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:37.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:36 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:36.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:37.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:36.646+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:37.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:36.646+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:37.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:36 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:36.646+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:37.060+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:37.060+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:37.060+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:36.910+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:36.910+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:37.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:36 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:36.910+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:37.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:37.237+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:37.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:37.237+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:37.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:37.237+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:37.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:37.237+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:07.698414+0000 front 2024-09-16T01:28:07.698390+0000 (oldest deadline 2024-09-16T01:28:36.498875+0000) 2024-09-16T01:28:37.617 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:37.210+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:37.617 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:37.210+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:37.617 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:37.210+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:37.617 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:37.211+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:37.889 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:37.616+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:37.889 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:37.616+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:37.889 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:37.616+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:38.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:37.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:38.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:37.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:38.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:37.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:38.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:37 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:37.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:38.213 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:38.078+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:38.213 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:38.078+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:38.213 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:38.078+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:38.214 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:37.888+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:38.214 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:37.888+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:38.214 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:37 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:37.888+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:38.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:38.229+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:47.195666+0000 front 2024-09-16T01:27:47.195717+0000 (oldest deadline 2024-09-16T01:28:10.695881+0000) 2024-09-16T01:28:38.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:38.229+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.796988+0000 front 2024-09-16T01:27:54.796906+0000 (oldest deadline 2024-09-16T01:28:19.496880+0000) 2024-09-16T01:28:38.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:38.229+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.998198+0000 front 2024-09-16T01:28:02.998006+0000 (oldest deadline 2024-09-16T01:28:27.697916+0000) 2024-09-16T01:28:38.529 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6[58564]: 2024-09-16T01:28:38.229+0000 7f2ac6598640 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:07.698414+0000 front 2024-09-16T01:28:07.698390+0000 (oldest deadline 2024-09-16T01:28:36.498875+0000) 2024-09-16T01:28:38.635 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:38.255+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:38.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:38.255+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:38.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:38.255+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:38.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:38.255+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:39.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:38.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:39.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:38.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:39.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:38.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:39.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:38 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:38.723+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:39.135 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:38.659+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:39.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:38.659+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:39.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:38 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:38.659+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:39.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:38.885+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:39.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:38.885+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:39.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:38 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:38.885+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:39.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:39.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:39.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:39.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:39.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:39.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:39.528 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:39 smithi078 podman[105164]: 2024-09-16 01:28:39.223493799 +0000 UTC m=+5.861723909 container died 557d18ee422f36d25d88ecf58383751f8bf40f6e901ef053c6c2969f82fa3a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True) 2024-09-16T01:28:39.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:39.260+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:39.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:39.260+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:39.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:39.260+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:39.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:39.260+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:39 smithi078 podman[105164]: 2024-09-16 01:28:39.634439397 +0000 UTC m=+6.272669496 container cleanup 557d18ee422f36d25d88ecf58383751f8bf40f6e901ef053c6c2969f82fa3a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:39 smithi078 bash[105164]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:39.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:39.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:39.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:39.853 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:39 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:39.712+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:40.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:39.692+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:40.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:39.692+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:40.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:39 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:39.692+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:40.160 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:39 smithi078 podman[105179]: 2024-09-16 01:28:39.852364393 +0000 UTC m=+0.628768268 container remove 557d18ee422f36d25d88ecf58383751f8bf40f6e901ef053c6c2969f82fa3a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:40.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:40.053+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:40.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:40.053+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:40.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:40.053+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:40.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:39.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:40.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:39.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:40.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:39 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:39.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:40.456 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.062537804 +0000 UTC m=+0.015438433 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:40.456 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.161392277 +0000 UTC m=+0.114292905 container create 8c7f3d8bbb91e986e8b9dd6412fe80a31cd744ac01bd3ca44aca0a09a744c90c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:40.456 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.380483667 +0000 UTC m=+0.333384295 container init 8c7f3d8bbb91e986e8b9dd6412fe80a31cd744ac01bd3ca44aca0a09a744c90c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:28:40.456 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.384102051 +0000 UTC m=+0.337002684 container start 8c7f3d8bbb91e986e8b9dd6412fe80a31cd744ac01bd3ca44aca0a09a744c90c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T01:28:40.456 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.455413292 +0000 UTC m=+0.408313936 container attach 8c7f3d8bbb91e986e8b9dd6412fe80a31cd744ac01bd3ca44aca0a09a744c90c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-16T01:28:40.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:40.233+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:40.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:40.233+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:40.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:40.233+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:40.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:40.233+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:40.778 INFO:journalctl@ceph.osd.6.smithi078.stdout:Sep 16 01:28:40 smithi078 podman[105268]: 2024-09-16 01:28:40.572652171 +0000 UTC m=+0.525552806 container died 8c7f3d8bbb91e986e8b9dd6412fe80a31cd744ac01bd3ca44aca0a09a744c90c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T01:28:40.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:40 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:40.665+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:40.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:40 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:40.665+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:40.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:40 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:40.665+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:40.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:40 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:40.665+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:41.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:40.667+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:41.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:40.667+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:41.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:40.667+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:41.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.6.service' 2024-09-16T01:28:41.287 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:41.288 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-16T01:28:41.288 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-16T01:28:41.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.7 2024-09-16T01:28:41.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:40.835+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:41.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:40.835+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:41.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:40 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:40.835+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:41.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:41.033+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:41.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:41.033+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:41.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:41.033+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:41.601 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 systemd[1]: Stopping Ceph osd.7 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:41.602 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:41.234+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:41.602 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:41.234+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:41.602 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:41.235+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:41.602 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:41.235+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:41.778 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:41 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:41.680+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:41.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:41 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:41.680+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:41.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:41 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:41.680+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:41.779 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:41 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:41.680+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:41.876 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.600+0000 7f9fb7bd9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:41.876 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.600+0000 7f9fb7bd9640 -1 osd.7 121 *** Got signal Terminated *** 2024-09-16T01:28:41.876 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.600+0000 7f9fb7bd9640 -1 osd.7 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:41.877 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.676+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:41.877 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.676+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:41.877 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:41 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:41.676+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:42.070 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:41.802+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:42.070 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:41.802+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:42.070 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:41 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:41.802+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:42.546 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:42.069+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:42.546 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:42.069+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:42.546 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:42.069+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:42.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:42.204+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:42.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:42.204+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:42.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:42.204+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:42.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:42.204+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:43.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:42.718+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:43.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:42.718+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:43.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:42.718+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:43.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:42 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:42.718+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:43.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:42.785+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:43.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:42.785+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:43.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:42 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:42.785+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:43.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.023+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:43.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.023+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:43.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.023+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:43.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:42.722+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:43.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:42.722+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:43.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:42.722+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:43.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:42 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:42.722+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.360952+0000 front 2024-09-16T01:28:20.360914+0000 (oldest deadline 2024-09-16T01:28:42.660956+0000) 2024-09-16T01:28:43.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:43.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:43.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:43.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:43.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:43.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:43.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:43.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:44.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:43.764+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:44.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:43.764+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:44.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:43.764+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:44.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:43.764+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:44.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:43 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:43.764+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:44.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:43.773+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:44.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:43.773+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:44.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:43 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:43.773+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:44.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.999+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:44.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.999+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:44.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.999+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:44.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:43.999+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:44.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:43.696+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:44.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:43.696+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:44.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:43.696+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:44.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:43 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:43.696+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.360952+0000 front 2024-09-16T01:28:20.360914+0000 (oldest deadline 2024-09-16T01:28:42.660956+0000) 2024-09-16T01:28:44.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:44.176+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:44.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:44.176+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:44.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:44.176+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:44.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:44.176+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:45.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:44.759+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:45.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:44.759+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:45.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:44.759+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:45.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:44.759+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:45.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:44 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:44.759+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:45.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:44.989+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:45.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:44.989+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:45.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:44.989+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:45.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:44.989+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:45.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:44.744+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:45.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:44.744+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:45.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:44.744+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:45.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:44 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:44.744+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:45.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:44.666+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:45.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:44.666+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:45.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:44.666+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:45.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:44.666+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.360952+0000 front 2024-09-16T01:28:20.360914+0000 (oldest deadline 2024-09-16T01:28:42.660956+0000) 2024-09-16T01:28:45.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:45.197+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:45.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:45.197+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:45.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:45.197+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:45.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:45.197+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:46.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:45.742+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:46.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:45.742+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:46.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:45.742+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:46.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:45.742+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:46.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:45 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:45.742+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:46.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:45.731+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:46.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:45.731+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:46.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:45.731+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:46.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:45.731+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:46.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:45.976+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:46.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:45.976+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:46.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:45.976+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:46.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:45 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:45.976+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:46.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:45.642+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.856885+0000 front 2024-09-16T01:27:55.857035+0000 (oldest deadline 2024-09-16T01:28:18.157050+0000) 2024-09-16T01:28:46.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:45.642+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:01.558992+0000 front 2024-09-16T01:28:01.558792+0000 (oldest deadline 2024-09-16T01:28:28.559553+0000) 2024-09-16T01:28:46.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:45.642+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.560049+0000 front 2024-09-16T01:28:08.560062+0000 (oldest deadline 2024-09-16T01:28:34.460014+0000) 2024-09-16T01:28:46.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:45 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7[52556]: 2024-09-16T01:28:45.642+0000 7f9fb01e7640 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.360952+0000 front 2024-09-16T01:28:20.360914+0000 (oldest deadline 2024-09-16T01:28:42.660956+0000) 2024-09-16T01:28:46.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:46.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:46.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:46.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:46.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:46.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:46.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:46.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:46.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:46 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:46.208+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:47.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:46.766+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:47.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:46.766+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:47.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:46.766+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:47.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:46.766+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:47.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:46 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:46.766+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:47.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:46.768+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:47.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:46.768+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:47.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:46.768+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:47.046 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:46.768+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:47.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:46.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:47.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:46.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:47.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:46.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:47.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:46 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:46.979+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:47.636 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:47 smithi148 podman[71832]: 2024-09-16 01:28:47.33086654 +0000 UTC m=+5.848645143 container died 604992f5b5816a25fcb9fc9acefa0a880a88653a8c66e473da678652ec21dadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:28:47.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:47 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:47.209+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:47.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:47 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:47.209+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:47.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:47 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:47.209+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:47.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:47 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:47.209+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:47.637 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:47 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:47.209+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:48.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:48.013+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:48.046 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:48.013+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:48.013+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:48.013+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:47 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:47.788+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:47 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:47.788+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:47 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:47.788+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:48.047 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:47 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:47.788+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:48.086 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:47 smithi148 podman[71832]: 2024-09-16 01:28:47.833730667 +0000 UTC m=+6.351509266 container cleanup 604992f5b5816a25fcb9fc9acefa0a880a88653a8c66e473da678652ec21dadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-16T01:28:48.086 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:47 smithi148 bash[71832]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7 2024-09-16T01:28:48.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:47.804+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:48.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:47.804+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:48.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:47.804+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:48.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:47.804+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:48.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:47 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:47.804+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:48.385 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71860]: 2024-09-16 01:28:48.085099886 +0000 UTC m=+0.755169737 container remove 604992f5b5816a25fcb9fc9acefa0a880a88653a8c66e473da678652ec21dadb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-16T01:28:48.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:48.183+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:48.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:48.183+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:48.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:48.183+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:48.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:48.183+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:48.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:48.183+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:48.797 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.409930624 +0000 UTC m=+0.015956571 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:48.797 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.535990546 +0000 UTC m=+0.142016480 container create 4708f73553e52898a1f50840928421cb1152cad4d9b68f5d79ec23cc2bac51ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-16T01:28:49.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.796689076 +0000 UTC m=+0.402715022 container init 4708f73553e52898a1f50840928421cb1152cad4d9b68f5d79ec23cc2bac51ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-16T01:28:49.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.80028892 +0000 UTC m=+0.406314867 container start 4708f73553e52898a1f50840928421cb1152cad4d9b68f5d79ec23cc2bac51ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-16T01:28:49.136 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.896846344 +0000 UTC m=+0.502872278 container attach 4708f73553e52898a1f50840928421cb1152cad4d9b68f5d79ec23cc2bac51ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T01:28:49.137 INFO:journalctl@ceph.osd.7.smithi148.stdout:Sep 16 01:28:48 smithi148 podman[71947]: 2024-09-16 01:28:48.986991557 +0000 UTC m=+0.593017492 container died 4708f73553e52898a1f50840928421cb1152cad4d9b68f5d79ec23cc2bac51ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-16T01:28:49.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:48.840+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:49.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:48.840+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:49.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:48.840+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:49.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:48.840+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:49.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:48 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:48.840+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:49.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:49.052+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:49.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:49.052+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:49.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:49.052+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:49.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:49.052+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:49.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:48.817+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:49.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:48.817+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:49.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:48.817+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:49.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:48 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:48.817+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:49.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:49.201+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:49.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:49.201+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:49.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:49.201+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:49.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:49.201+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:49.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:49 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:49.201+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:49.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.7.service' 2024-09-16T01:28:49.715 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:49.715 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-16T01:28:49.715 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-16T01:28:49.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.8 2024-09-16T01:28:50.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:49.858+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:50.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:49.858+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:50.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:49.858+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:50.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:49 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:49.858+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:49 smithi193 systemd[1]: Stopping Ceph osd.8 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.007+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.007+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.007+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.007+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.009+0000 7f15faea2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.009+0000 7f15faea2640 -1 osd.8 121 *** Got signal Terminated *** 2024-09-16T01:28:50.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:50.009+0000 7f15faea2640 -1 osd.8 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:50.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:49 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:49.852+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:50.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:49 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:49.852+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:50.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:49 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:49.852+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:50.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:49 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:49.852+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:50.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:49 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:49.852+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:50.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:50 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:50.180+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:50.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:50 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:50.180+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:50.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:50 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:50.180+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:50.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:50 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:50.180+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:50.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:50 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:50.180+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:51.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:50 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:50.874+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:51.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:50 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:50.874+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:51.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:50 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:50.874+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:51.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:50 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:50.874+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:51.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:50 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:50.874+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:51.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:50.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:51.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:50.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:51.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:50.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:51.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:50 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:50.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:51.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:51.056+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:51.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:51.056+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:51.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:51.056+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:51.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:51.056+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:51.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:51.144+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:51.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:51.144+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:51.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:51.144+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:51.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:51.144+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:51.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:51 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:51.144+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:52.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:51 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:51.832+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:52.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:51 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:51.832+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:52.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:51 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:51.832+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:52.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:51 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:51.832+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:52.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:51 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:51.832+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:52.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:51.916+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:52.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:51.916+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:52.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:51.916+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:52.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:51 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:51.916+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:52.296 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:52.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:52.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:52.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:52.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:52.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:52.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:52.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:52.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:52.073+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.323164+0000 front 2024-09-16T01:28:28.323203+0000 (oldest deadline 2024-09-16T01:28:51.223366+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:52.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:52 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:52.139+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:53.029 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:52.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:53.033 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:52.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:53.033 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:52.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:53.034 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:52.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:53.034 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:52 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:52.793+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:53.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:52.940+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:52.940+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:52.940+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:52 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:52.940+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:53.105+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:53.105+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:53.105+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:53.105+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:53.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:53.105+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.323164+0000 front 2024-09-16T01:28:28.323203+0000 (oldest deadline 2024-09-16T01:28:51.223366+0000) 2024-09-16T01:28:53.385 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:53.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:53.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:53.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:53.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:53.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:53 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:53.125+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:54.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:53.813+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:54.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:53.813+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:54.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:53.813+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:54.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:53.813+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:54.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:53 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:53.813+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:54.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:53.912+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:54.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:53.912+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:54.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:53.912+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:53.912+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:54.099+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.019251+0000 front 2024-09-16T01:27:55.018968+0000 (oldest deadline 2024-09-16T01:28:20.318934+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:54.099+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:00.319212+0000 front 2024-09-16T01:28:00.319231+0000 (oldest deadline 2024-09-16T01:28:28.519914+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:54.099+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:10.820820+0000 front 2024-09-16T01:28:10.820881+0000 (oldest deadline 2024-09-16T01:28:36.120735+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:54.099+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.822031+0000 front 2024-09-16T01:28:20.822013+0000 (oldest deadline 2024-09-16T01:28:43.622393+0000) 2024-09-16T01:28:54.297 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8[54039]: 2024-09-16T01:28:54.099+0000 7f15f74b8640 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.323164+0000 front 2024-09-16T01:28:28.323203+0000 (oldest deadline 2024-09-16T01:28:51.223366+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:54.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:54 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:54.140+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:55.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:54.823+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:55.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:54.823+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:55.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:54.823+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:55.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:54.823+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:55.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:54 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:54.823+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:55.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:54.897+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:55.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:54.897+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:55.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:54.897+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:55.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:54 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:54.897+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:55.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:55 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:55.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:55.910 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:55 smithi193 podman[73922]: 2024-09-16 01:28:55.547698482 +0000 UTC m=+5.640474636 container died 049f75ad26b4deec318c7b84ba5e34205e77aa3fdc21619f538e06eceeb7fddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-16T01:28:56.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[73922]: 2024-09-16 01:28:56.051194758 +0000 UTC m=+6.143970919 container cleanup 049f75ad26b4deec318c7b84ba5e34205e77aa3fdc21619f538e06eceeb7fddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-16T01:28:56.253 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 bash[73922]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8 2024-09-16T01:28:56.254 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:55 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:55.909+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:56.254 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:55 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:55.909+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:56.254 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:55 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:55.909+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:56.254 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:55 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:55.909+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:56.254 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:55 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:55.909+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:28:56.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:56.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:56.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:56.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:56.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:56.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:55 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:55.787+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:28:56.546 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[73938]: 2024-09-16 01:28:56.253007394 +0000 UTC m=+0.696295490 container remove 049f75ad26b4deec318c7b84ba5e34205e77aa3fdc21619f538e06eceeb7fddf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:56.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:56 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:56.196+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:56.816 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[74026]: 2024-09-16 01:28:56.465174389 +0000 UTC m=+0.016300135 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:28:56.816 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[74026]: 2024-09-16 01:28:56.587335397 +0000 UTC m=+0.138461144 container create fc95b11fc23f32aa9478819b7a1cdef5a736a53ee2a4c9d347765e705bf82415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-16T01:28:56.816 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[74026]: 2024-09-16 01:28:56.815054469 +0000 UTC m=+0.366180218 container init fc95b11fc23f32aa9478819b7a1cdef5a736a53ee2a4c9d347765e705bf82415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e) 2024-09-16T01:28:57.269 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[74026]: 2024-09-16 01:28:56.818956863 +0000 UTC m=+0.370082610 container start fc95b11fc23f32aa9478819b7a1cdef5a736a53ee2a4c9d347765e705bf82415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:28:57.269 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:56 smithi193 podman[74026]: 2024-09-16 01:28:56.931577599 +0000 UTC m=+0.482703346 container attach fc95b11fc23f32aa9478819b7a1cdef5a736a53ee2a4c9d347765e705bf82415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8-deactivate, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:28:57.269 INFO:journalctl@ceph.osd.8.smithi193.stdout:Sep 16 01:28:57 smithi193 podman[74026]: 2024-09-16 01:28:57.003457875 +0000 UTC m=+0.554583624 container died fc95b11fc23f32aa9478819b7a1cdef5a736a53ee2a4c9d347765e705bf82415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-16T01:28:57.270 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:56.903+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:57.270 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:56.903+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:57.270 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:56.903+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:57.270 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:56.903+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:57.270 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:56.903+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:28:57.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:57.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:57.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:57.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:57.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:57.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:56 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:56.801+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:28:57.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.8.service' 2024-09-16T01:28:57.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:28:57.624 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-16T01:28:57.624 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-16T01:28:57.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.9 2024-09-16T01:28:57.635 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:57.635 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:57.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:57.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:57.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:57.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:57 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:57.244+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:57.938 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 systemd[1]: Stopping Ceph osd.9 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.937+0000 7fef3cddc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.937+0000 7fef3cddc640 -1 osd.9 121 *** Got signal Terminated *** 2024-09-16T01:28:57.939 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:57 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:57.937+0000 7fef3cddc640 -1 osd.9 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:28:58.241 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:57 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:57.853+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:58.241 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:57 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:57.853+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:58.241 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:57 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:57.853+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:58.241 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:57 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:57.853+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:58.241 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:57 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:57.853+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:58.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:58 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:58.205+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:28:59.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:28:59.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:28:59.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:28:59.280 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:28:59.280 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:28:59.280 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:58 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:58.790+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:28:59.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:58 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:58.884+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:28:59.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:58 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:58.884+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:28:59.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:58 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:58.884+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:28:59.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:58 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:58.884+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:28:59.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:58 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:58.884+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:28:59.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:28:59 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:28:59.173+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:00.133 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:00.133 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:00.133 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:00.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:29:00.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:29:00.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:29:00.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:29:00.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:29:00.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:28:59 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:28:59.838+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:29:00.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:59.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:00.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:59.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:00.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:59.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:00.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:59.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:00.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:28:59 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:28:59.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:00.385 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:00.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:00.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:00 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:00.131+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:29:01.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:00 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:00.814+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:29:01.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:00.894+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:01.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:00.894+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:01.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:00.894+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:01.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:00.894+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:01.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:00.894+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:01.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:01.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:01.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:01.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:01.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:01.637 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:01.637 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:01 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:01.151+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:29:02.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:01 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:01.782+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:29:02.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:02.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:02.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:02.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:02.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:02.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:01 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:01.925+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:02.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:02 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:02.128+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:03.278 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:43.957560+0000 front 2024-09-16T01:27:43.957537+0000 (oldest deadline 2024-09-16T01:28:08.657696+0000) 2024-09-16T01:29:03.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:55.559693+0000 front 2024-09-16T01:27:55.559719+0000 (oldest deadline 2024-09-16T01:28:17.259741+0000) 2024-09-16T01:29:03.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:02.560890+0000 front 2024-09-16T01:28:02.560958+0000 (oldest deadline 2024-09-16T01:28:28.460902+0000) 2024-09-16T01:29:03.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.461368+0000 front 2024-09-16T01:28:08.461304+0000 (oldest deadline 2024-09-16T01:28:37.261759+0000) 2024-09-16T01:29:03.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.762644+0000 front 2024-09-16T01:28:20.762709+0000 (oldest deadline 2024-09-16T01:28:43.662751+0000) 2024-09-16T01:29:03.279 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:02 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9[65084]: 2024-09-16T01:29:02.809+0000 7fef353ea640 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:28.363815+0000 front 2024-09-16T01:28:28.363837+0000 (oldest deadline 2024-09-16T01:28:55.363898+0000) 2024-09-16T01:29:03.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:03.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:03.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:03.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:03.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:03.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:02 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:02.895+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:03.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:03 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:03.132+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:04.221 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:03 smithi078 podman[105382]: 2024-09-16 01:29:03.963161649 +0000 UTC m=+6.144262141 container died d0841ad4c88d0035a3823c18d950490dbff8e9701eecf5c15910a1fd303d5642 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:04.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:04.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:04.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:04.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:04.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:04.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:03 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:03.898+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:04.528 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:04 smithi078 podman[105382]: 2024-09-16 01:29:04.437298224 +0000 UTC m=+6.618398764 container cleanup d0841ad4c88d0035a3823c18d950490dbff8e9701eecf5c15910a1fd303d5642 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-16T01:29:04.529 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:04 smithi078 bash[105382]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:04.636 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:04 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:04.141+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:05.028 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:04 smithi078 podman[105398]: 2024-09-16 01:29:04.738967539 +0000 UTC m=+0.772080700 container remove d0841ad4c88d0035a3823c18d950490dbff8e9701eecf5c15910a1fd303d5642 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS) 2024-09-16T01:29:05.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:05.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:05.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:05.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:05.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:05.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:04.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:05.301 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:04.950021137 +0000 UTC m=+0.015420812 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:29:05.301 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:05.064770045 +0000 UTC m=+0.130169702 container create e1d60dc65b910bd0f9fa723595120852d55cdff45a75417b724259211a1b4173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:05.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:05 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:05.108+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:05.762 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:05.300605527 +0000 UTC m=+0.366005190 container init e1d60dc65b910bd0f9fa723595120852d55cdff45a75417b724259211a1b4173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:29:05.763 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:05.303990406 +0000 UTC m=+0.369390074 container start e1d60dc65b910bd0f9fa723595120852d55cdff45a75417b724259211a1b4173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-16T01:29:05.763 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:05.375450463 +0000 UTC m=+0.440850124 container attach e1d60dc65b910bd0f9fa723595120852d55cdff45a75417b724259211a1b4173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:29:05.763 INFO:journalctl@ceph.osd.9.smithi078.stdout:Sep 16 01:29:05 smithi078 podman[105483]: 2024-09-16 01:29:05.488394104 +0000 UTC m=+0.553793764 container died e1d60dc65b910bd0f9fa723595120852d55cdff45a75417b724259211a1b4173 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-16T01:29:06.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.9.service' 2024-09-16T01:29:06.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:06.118 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-16T01:29:06.118 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-16T01:29:06.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.10 2024-09-16T01:29:06.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:06.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:06.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:06.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:06.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:06.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:05 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:05.867+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.086+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:06.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 systemd[1]: Stopping Ceph osd.10 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:06.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.393+0000 7fe9f35d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:06.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.393+0000 7fe9f35d9640 -1 osd.10 121 *** Got signal Terminated *** 2024-09-16T01:29:06.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:06 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:06.393+0000 7fe9f35d9640 -1 osd.10 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:07.198 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:06 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:06.896+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:07.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.038+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:08.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:08.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:08.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:08.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:08.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:08.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:07 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:07.871+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:08.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:07 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:07.988+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:09.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:09.306 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:08 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:08.915+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:09.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:09 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:09.006+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:10.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:10.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:10.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:10.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:10.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:10.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:10.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:09 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:09.934+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:10.386 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:10.387 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.013+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:11.287 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.809672+0000 front 2024-09-16T01:27:46.809647+0000 (oldest deadline 2024-09-16T01:28:10.909653+0000) 2024-09-16T01:29:11.287 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6804 osd.1 since back 2024-09-16T01:27:54.410710+0000 front 2024-09-16T01:27:54.410673+0000 (oldest deadline 2024-09-16T01:28:19.011403+0000) 2024-09-16T01:29:11.287 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:03.712612+0000 front 2024-09-16T01:28:03.712547+0000 (oldest deadline 2024-09-16T01:28:27.212467+0000) 2024-09-16T01:29:11.287 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:12.314318+0000 front 2024-09-16T01:28:12.314462+0000 (oldest deadline 2024-09-16T01:28:35.214390+0000) 2024-09-16T01:29:11.288 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:20.415770+0000 front 2024-09-16T01:28:20.415735+0000 (oldest deadline 2024-09-16T01:28:45.715888+0000) 2024-09-16T01:29:11.288 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:27.417011+0000 front 2024-09-16T01:28:27.417060+0000 (oldest deadline 2024-09-16T01:28:51.417220+0000) 2024-09-16T01:29:11.288 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:10 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10[59278]: 2024-09-16T01:29:10.964+0000 7fe9ebbe7640 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:37.218715+0000 front 2024-09-16T01:28:37.218774+0000 (oldest deadline 2024-09-16T01:29:00.718625+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:11.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:11.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:10 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:10.941+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:12.254 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:11 smithi148 podman[72051]: 2024-09-16 01:29:11.992251154 +0000 UTC m=+5.677730349 container died e65ec22035c5f33c22769ba758954984c6410de8e982ac28573ddee0be7cfc59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:29:12.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:12.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:12.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:12.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:12.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:12.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:12.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:11 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:11.961+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:12.504 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:12 smithi148 podman[72051]: 2024-09-16 01:29:12.335459379 +0000 UTC m=+6.020938571 container cleanup e65ec22035c5f33c22769ba758954984c6410de8e982ac28573ddee0be7cfc59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-16T01:29:12.504 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:12 smithi148 bash[72051]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10 2024-09-16T01:29:12.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:12 smithi148 podman[72066]: 2024-09-16 01:29:12.503534801 +0000 UTC m=+0.507945165 container remove e65ec22035c5f33c22769ba758954984c6410de8e982ac28573ddee0be7cfc59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-16T01:29:13.165 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:12 smithi148 podman[72157]: 2024-09-16 01:29:12.895936662 +0000 UTC m=+0.113238100 container create 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:29:13.166 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:12 smithi148 podman[72157]: 2024-09-16 01:29:12.798807052 +0000 UTC m=+0.016108503 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:29:13.166 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 podman[72157]: 2024-09-16 01:29:13.090091634 +0000 UTC m=+0.307393079 container init 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-16T01:29:13.166 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 podman[72157]: 2024-09-16 01:29:13.094018664 +0000 UTC m=+0.311320103 container start 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:29:13.166 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 podman[72157]: 2024-09-16 01:29:13.164816967 +0000 UTC m=+0.382118417 container attach 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-16T01:29:13.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:13.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:13.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:13.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:13.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:13.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:13.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:12 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:12.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:13.618 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 podman[72157]: 2024-09-16 01:29:13.280117228 +0000 UTC m=+0.497418669 container died 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:13.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.10.service' 2024-09-16T01:29:13.885 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 podman[72180]: 2024-09-16 01:29:13.851209256 +0000 UTC m=+0.560995729 container remove 8f134068487931073824f8efdaea2d942df5182feff3445262fee0b017e76e87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=) 2024-09-16T01:29:13.885 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.10.service: Deactivated successfully. 2024-09-16T01:29:13.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 systemd[1]: Stopped Ceph osd.10 for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:29:13.886 INFO:journalctl@ceph.osd.10.smithi148.stdout:Sep 16 01:29:13 smithi148 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.10.service: Consumed 7min 4.130s CPU time. 2024-09-16T01:29:13.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:13.907 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-16T01:29:13.907 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-16T01:29:13.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.11 2024-09-16T01:29:14.281 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 systemd[1]: Stopping Ceph osd.11 for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:14.281 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:14.281 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:13 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:13.986+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:14 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:14.278+0000 7f8625908640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:14 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:14.278+0000 7f8625908640 -1 osd.11 121 *** Got signal Terminated *** 2024-09-16T01:29:14.282 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:14 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:14.278+0000 7f8625908640 -1 osd.11 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-16T01:29:15.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:15.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.018+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:16.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:16.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:16.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:16.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:16.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:16.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:16.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:15 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:15.972+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:17.259 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:17.260 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:16 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:16.935+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:18.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:18.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:17 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:17.944+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:19.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-09-16T01:27:46.819120+0000 front 2024-09-16T01:27:46.819417+0000 (oldest deadline 2024-09-16T01:28:09.619703+0000) 2024-09-16T01:29:19.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-09-16T01:28:04.722729+0000 front 2024-09-16T01:28:04.722517+0000 (oldest deadline 2024-09-16T01:28:27.622681+0000) 2024-09-16T01:29:19.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-09-16T01:28:08.723513+0000 front 2024-09-16T01:28:08.723706+0000 (oldest deadline 2024-09-16T01:28:34.023567+0000) 2024-09-16T01:29:19.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6812 osd.4 since back 2024-09-16T01:28:18.625024+0000 front 2024-09-16T01:28:18.625043+0000 (oldest deadline 2024-09-16T01:28:43.924963+0000) 2024-09-16T01:29:19.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-09-16T01:28:29.826135+0000 front 2024-09-16T01:28:29.826064+0000 (oldest deadline 2024-09-16T01:28:55.726171+0000) 2024-09-16T01:29:19.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-09-16T01:28:36.227273+0000 front 2024-09-16T01:28:36.227299+0000 (oldest deadline 2024-09-16T01:29:01.527280+0000) 2024-09-16T01:29:19.297 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:18 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11[60991]: 2024-09-16T01:29:18.927+0000 7f8621f1e640 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.148:6820 osd.7 since back 2024-09-16T01:28:44.928934+0000 front 2024-09-16T01:28:44.928897+0000 (oldest deadline 2024-09-16T01:29:08.428889+0000) 2024-09-16T01:29:20.181 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:19 smithi193 podman[74140]: 2024-09-16 01:29:19.910250702 +0000 UTC m=+5.806428576 container died de614862893fecfa7e2a0e9fb6d0a5df7b009229667852367ef3ab63ef2e693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-16T01:29:20.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:20 smithi193 podman[74140]: 2024-09-16 01:29:20.262909242 +0000 UTC m=+6.159087157 container cleanup de614862893fecfa7e2a0e9fb6d0a5df7b009229667852367ef3ab63ef2e693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:29:20.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:20 smithi193 bash[74140]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11 2024-09-16T01:29:20.874 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:20 smithi193 podman[74156]: 2024-09-16 01:29:20.547519971 +0000 UTC m=+0.636034117 container remove de614862893fecfa7e2a0e9fb6d0a5df7b009229667852367ef3ab63ef2e693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731) 2024-09-16T01:29:21.160 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:20 smithi193 podman[74244]: 2024-09-16 01:29:20.873654595 +0000 UTC m=+0.108790532 container create 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:21.160 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:20 smithi193 podman[74244]: 2024-09-16 01:29:20.781025023 +0000 UTC m=+0.016160911 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-16T01:29:21.160 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:21 smithi193 podman[74244]: 2024-09-16 01:29:21.084354982 +0000 UTC m=+0.319490863 container init 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T01:29:21.160 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:21 smithi193 podman[74244]: 2024-09-16 01:29:21.087654122 +0000 UTC m=+0.322790012 container start 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD) 2024-09-16T01:29:21.160 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:21 smithi193 podman[74244]: 2024-09-16 01:29:21.159199164 +0000 UTC m=+0.394335047 container attach 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-16T01:29:21.546 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:21 smithi193 podman[74244]: 2024-09-16 01:29:21.27421421 +0000 UTC m=+0.509350088 container died 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:29:21.796 INFO:journalctl@ceph.osd.11.smithi193.stdout:Sep 16 01:29:21 smithi193 podman[74267]: 2024-09-16 01:29:21.779176816 +0000 UTC m=+0.495564778 container remove 9ff54d58e654547cc2e7d8661e86c4328fce58a48443874826ffc872f2344ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-16T01:29:21.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@osd.11.service' 2024-09-16T01:29:21.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:21.841 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-16T01:29:21.841 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-16T01:29:21.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.a 2024-09-16T01:29:22.156 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:21 smithi078 systemd[1]: Stopping Ceph mds.a for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:22.418 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a[71950]: 2024-09-16T01:29:22.155+0000 7f75bed54640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:22.418 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:22 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a[71950]: 2024-09-16T01:29:22.155+0000 7f75bed54640 -1 mds.a *** got signal Terminated *** 2024-09-16T01:29:23.432 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:23 smithi078 podman[105586]: 2024-09-16 01:29:23.142253753 +0000 UTC m=+1.106755848 container died 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-8921c8e) 2024-09-16T01:29:23.778 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:23 smithi078 podman[105586]: 2024-09-16 01:29:23.53892324 +0000 UTC m=+1.503425335 container cleanup 1b8d9272c81f7e0152dbb03f15a1d73853a0764eae35501f77e3bafc579e29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:23.778 INFO:journalctl@ceph.mds.a.smithi078.stdout:Sep 16 01:29:23 smithi078 bash[105586]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-a 2024-09-16T01:29:23.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.a.service' 2024-09-16T01:29:23.965 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:23.965 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-16T01:29:23.965 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-16T01:29:23.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.d 2024-09-16T01:29:24.279 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:24 smithi078 systemd[1]: Stopping Ceph mds.d for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:24.778 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d[70765]: 2024-09-16T01:29:24.326+0000 7f0db2767640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:24.779 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:24 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d[70765]: 2024-09-16T01:29:24.326+0000 7f0db2767640 -1 mds.d *** got signal Terminated *** 2024-09-16T01:29:28.528 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:28 smithi078 podman[105719]: 2024-09-16 01:29:28.228667651 +0000 UTC m=+4.037281696 container died 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:29:28.978 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:28 smithi078 podman[105719]: 2024-09-16 01:29:28.717870926 +0000 UTC m=+4.526484973 container cleanup 8a40bcc7f2b402ef745a5ea8c8dbb9d439d8ab566ca1164deb6b8b8a853efc58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:29:28.978 INFO:journalctl@ceph.mds.d.smithi078.stdout:Sep 16 01:29:28 smithi078 bash[105719]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-d 2024-09-16T01:29:29.099 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.d.service' 2024-09-16T01:29:29.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:29.134 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-16T01:29:29.134 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-16T01:29:29.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.g 2024-09-16T01:29:29.278 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:29 smithi078 systemd[1]: Stopping Ceph mds.g for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:29.778 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g[71564]: 2024-09-16T01:29:29.462+0000 7fae10575640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:29.779 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:29 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g[71564]: 2024-09-16T01:29:29.462+0000 7fae10575640 -1 mds.g *** got signal Terminated *** 2024-09-16T01:29:32.157 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:31 smithi078 podman[105839]: 2024-09-16 01:29:31.871547579 +0000 UTC m=+2.520915030 container died 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:29:32.503 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 podman[105839]: 2024-09-16 01:29:32.305505031 +0000 UTC m=+2.954872481 container cleanup 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-16T01:29:32.503 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 bash[105839]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g 2024-09-16T01:29:32.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.g.service' 2024-09-16T01:29:32.755 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 podman[105854]: 2024-09-16 01:29:32.50257632 +0000 UTC m=+0.630053388 container remove 14980425d88406036e7da397f2334268bd4f42b9c52a748c6c94d344897b7b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-g, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-16T01:29:32.755 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.g.service: Deactivated successfully. 2024-09-16T01:29:32.755 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 systemd[1]: Stopped Ceph mds.g for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:29:32.756 INFO:journalctl@ceph.mds.g.smithi078.stdout:Sep 16 01:29:32 smithi078 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.g.service: Consumed 1.108s CPU time. 2024-09-16T01:29:32.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:32.768 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-16T01:29:32.768 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-16T01:29:32.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.j 2024-09-16T01:29:33.028 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:32 smithi078 systemd[1]: Stopping Ceph mds.j for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:33.447 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j[70367]: 2024-09-16T01:29:33.091+0000 7fb47ea58640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:33.447 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:33 smithi078 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j[70367]: 2024-09-16T01:29:33.091+0000 7fb47ea58640 -1 mds.j *** got signal Terminated *** 2024-09-16T01:29:36.823 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:36 smithi078 podman[105963]: 2024-09-16 01:29:36.543744251 +0000 UTC m=+3.559856820 container died 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True) 2024-09-16T01:29:37.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:36 smithi078 podman[105963]: 2024-09-16 01:29:36.901106759 +0000 UTC m=+3.917219317 container cleanup 523cf01e8a22d906ebc898c1d3b1199e0557636e64d3dfd4591e12fa765d5ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:29:37.188 INFO:journalctl@ceph.mds.j.smithi078.stdout:Sep 16 01:29:36 smithi078 bash[105963]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-j 2024-09-16T01:29:37.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.j.service' 2024-09-16T01:29:37.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:37.340 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-16T01:29:37.340 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-16T01:29:37.340 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.b 2024-09-16T01:29:37.627 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:37 smithi148 systemd[1]: Stopping Ceph mds.b for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:37.886 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: 2024-09-16T01:29:37.625+0000 7fb42ee8d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:37.886 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:37 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b[64473]: 2024-09-16T01:29:37.625+0000 7fb42ee8d640 -1 mds.b *** got signal Terminated *** 2024-09-16T01:29:39.635 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:39 smithi148 podman[72261]: 2024-09-16 01:29:39.367825679 +0000 UTC m=+1.840090938 container died 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-16T01:29:39.936 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:39 smithi148 podman[72261]: 2024-09-16 01:29:39.735694836 +0000 UTC m=+2.207960099 container cleanup 8f3a27f8eb92af2f548b260c16047de25943a9117763c6ca0cec65e8896c6069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-16T01:29:39.936 INFO:journalctl@ceph.mds.b.smithi148.stdout:Sep 16 01:29:39 smithi148 bash[72261]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-b 2024-09-16T01:29:40.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.b.service' 2024-09-16T01:29:40.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:40.191 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-16T01:29:40.191 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-16T01:29:40.191 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.e 2024-09-16T01:29:40.506 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:40 smithi148 systemd[1]: Stopping Ceph mds.e for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:40.885 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: 2024-09-16T01:29:40.503+0000 7f4cc5b27640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:40.886 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:40 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e[63869]: 2024-09-16T01:29:40.503+0000 7f4cc5b27640 -1 mds.e *** got signal Terminated *** 2024-09-16T01:29:43.636 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:43 smithi148 podman[72382]: 2024-09-16 01:29:43.373984183 +0000 UTC m=+2.967957294 container died 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-16T01:29:44.006 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:43 smithi148 podman[72382]: 2024-09-16 01:29:43.721433304 +0000 UTC m=+3.315406423 container cleanup 2d1c85e410e083553f06be1c5d7711d65e6109da3c10db6617b51270cdf3915a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-16T01:29:44.006 INFO:journalctl@ceph.mds.e.smithi148.stdout:Sep 16 01:29:43 smithi148 bash[72382]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-e 2024-09-16T01:29:44.151 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.e.service' 2024-09-16T01:29:44.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:44.184 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-16T01:29:44.184 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-16T01:29:44.184 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.h 2024-09-16T01:29:44.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:44 smithi148 systemd[1]: Stopping Ceph mds.h for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:44.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: 2024-09-16T01:29:44.491+0000 7f76d9cc4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:44.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:44 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h[64171]: 2024-09-16T01:29:44.491+0000 7f76d9cc4640 -1 mds.h *** got signal Terminated *** 2024-09-16T01:29:47.636 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:47 smithi148 podman[72504]: 2024-09-16 01:29:47.369861458 +0000 UTC m=+2.973705251 container died f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-16T01:29:48.093 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:47 smithi148 podman[72504]: 2024-09-16 01:29:47.916722509 +0000 UTC m=+3.520566291 container cleanup f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-16T01:29:48.093 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:47 smithi148 bash[72504]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h 2024-09-16T01:29:48.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.h.service' 2024-09-16T01:29:48.366 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:48 smithi148 podman[72519]: 2024-09-16 01:29:48.092886219 +0000 UTC m=+0.716893750 container remove f60a393650bb2b611db684eba3ee72dddf4eeb9afad96c5aff5b1c707611eeb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:48.366 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:48 smithi148 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.h.service: Deactivated successfully. 2024-09-16T01:29:48.366 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:48 smithi148 systemd[1]: Stopped Ceph mds.h for fa4731de-73c6-11ef-bceb-c7b262605968. 2024-09-16T01:29:48.366 INFO:journalctl@ceph.mds.h.smithi148.stdout:Sep 16 01:29:48 smithi148 systemd[1]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.h.service: Consumed 27.575s CPU time. 2024-09-16T01:29:48.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:48.377 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-16T01:29:48.377 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-16T01:29:48.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.k 2024-09-16T01:29:48.636 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:48 smithi148 systemd[1]: Stopping Ceph mds.k for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:49.136 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k[63554]: 2024-09-16T01:29:48.694+0000 7fa204814640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:49.136 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:48 smithi148 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k[63554]: 2024-09-16T01:29:48.694+0000 7fa204814640 -1 mds.k *** got signal Terminated *** 2024-09-16T01:29:53.096 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:52 smithi148 podman[72625]: 2024-09-16 01:29:52.701445268 +0000 UTC m=+4.107165252 container died 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=) 2024-09-16T01:29:53.355 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:53 smithi148 podman[72625]: 2024-09-16 01:29:53.178619498 +0000 UTC m=+4.584339477 container cleanup 891e4f2530e16cdb7d1433b7539cccbd9894dfe71e2d5892d92f92e770a8dd94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-16T01:29:53.355 INFO:journalctl@ceph.mds.k.smithi148.stdout:Sep 16 01:29:53 smithi148 bash[72625]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-k 2024-09-16T01:29:53.561 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.k.service' 2024-09-16T01:29:53.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:53.593 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-16T01:29:53.593 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-16T01:29:53.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.c 2024-09-16T01:29:53.989 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:53 smithi193 systemd[1]: Stopping Ceph mds.c for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:54.289 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: 2024-09-16T01:29:53.987+0000 7f5f64168640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:54.289 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:53 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c[66234]: 2024-09-16T01:29:53.987+0000 7f5f64168640 -1 mds.c *** got signal Terminated *** 2024-09-16T01:29:55.613 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:55 smithi193 podman[74347]: 2024-09-16 01:29:55.36207533 +0000 UTC m=+1.577069782 container died cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-16T01:29:55.997 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:55 smithi193 podman[74347]: 2024-09-16 01:29:55.711751303 +0000 UTC m=+1.926745744 container cleanup cb7b57758f1ec98449878a2a3f0d96477d6c30e2d74455507718aa3e20a89b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-16T01:29:55.997 INFO:journalctl@ceph.mds.c.smithi193.stdout:Sep 16 01:29:55 smithi193 bash[74347]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-c 2024-09-16T01:29:56.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.c.service' 2024-09-16T01:29:56.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:29:56.179 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-16T01:29:56.180 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-16T01:29:56.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.f 2024-09-16T01:29:56.546 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:56 smithi193 systemd[1]: Stopping Ceph mds.f for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:29:56.547 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: 2024-09-16T01:29:56.499+0000 7fc401a69640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:29:56.547 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:56 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f[65930]: 2024-09-16T01:29:56.499+0000 7fc401a69640 -1 mds.f *** got signal Terminated *** 2024-09-16T01:29:59.666 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:59 smithi193 podman[74480]: 2024-09-16 01:29:59.343427267 +0000 UTC m=+2.948914338 container died 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-16T01:29:59.666 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:59 smithi193 podman[74480]: 2024-09-16 01:29:59.665480762 +0000 UTC m=+3.270967831 container cleanup 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-16T01:30:00.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:59 smithi193 bash[74480]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f 2024-09-16T01:30:00.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Sep 16 01:29:59 smithi193 podman[74495]: 2024-09-16 01:29:59.85870344 +0000 UTC m=+0.510798064 container remove 3c2b91514f2474e61d75d9a59c513251b3aae09c59dcd6180c1e31128f94b413 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-16T01:30:00.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.f.service' 2024-09-16T01:30:00.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:30:00.112 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-16T01:30:00.112 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-16T01:30:00.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.i 2024-09-16T01:30:00.429 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:00 smithi193 systemd[1]: Stopping Ceph mds.i for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:30:00.796 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: 2024-09-16T01:30:00.427+0000 7fec47d1d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:30:00.796 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:00 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i[66532]: 2024-09-16T01:30:00.427+0000 7fec47d1d640 -1 mds.i *** got signal Terminated *** 2024-09-16T01:30:03.612 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:03 smithi193 podman[74602]: 2024-09-16 01:30:03.337529256 +0000 UTC m=+3.008253520 container died 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-16T01:30:03.988 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:03 smithi193 podman[74602]: 2024-09-16 01:30:03.702751023 +0000 UTC m=+3.373475287 container cleanup 1699aba69888184ff837d7a3c4fc0bece07368603ae3b1c7e90015784f395b50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-16T01:30:03.988 INFO:journalctl@ceph.mds.i.smithi193.stdout:Sep 16 01:30:03 smithi193 bash[74602]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-i 2024-09-16T01:30:04.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.i.service' 2024-09-16T01:30:04.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:30:04.173 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-16T01:30:04.173 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-16T01:30:04.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.l 2024-09-16T01:30:04.542 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:04 smithi193 systemd[1]: Stopping Ceph mds.l for fa4731de-73c6-11ef-bceb-c7b262605968... 2024-09-16T01:30:04.796 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l[65630]: 2024-09-16T01:30:04.548+0000 7fab903e6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-16T01:30:04.796 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:04 smithi193 ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l[65630]: 2024-09-16T01:30:04.548+0000 7fab903e6640 -1 mds.l *** got signal Terminated *** 2024-09-16T01:30:06.768 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:06 smithi193 podman[74726]: 2024-09-16 01:30:06.502329794 +0000 UTC m=+2.107718457 container died 5ac6618a7df0ba967d7d3554473f800141c4aa2cbbc68aaf67737e509cbaf272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-16T01:30:07.046 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:06 smithi193 podman[74726]: 2024-09-16 01:30:06.856884674 +0000 UTC m=+2.462273336 container cleanup 5ac6618a7df0ba967d7d3554473f800141c4aa2cbbc68aaf67737e509cbaf272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-16T01:30:07.046 INFO:journalctl@ceph.mds.l.smithi193.stdout:Sep 16 01:30:06 smithi193 bash[74726]: ceph-fa4731de-73c6-11ef-bceb-c7b262605968-mds-l 2024-09-16T01:30:07.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fa4731de-73c6-11ef-bceb-c7b262605968@mds.l.service' 2024-09-16T01:30:07.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-16T01:30:07.319 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-16T01:30:07.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T01:30:07.522 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:30:10.055 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T01:30:10.260 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:30:12.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-16T01:30:12.941 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:30:15.421 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:30:15.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:30:15.487 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-16T01:30:15.514 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-16T01:30:15.516 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi078/crash 2024-09-16T01:30:15.516 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash -- . 2024-09-16T01:30:15.575 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T01:30:15.576 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2024-09-16T01:30:15.578 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi148/crash 2024-09-16T01:30:15.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash -- . 2024-09-16T01:30:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:tar: /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T01:30:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:tar: Error is not recoverable: exiting now 2024-09-16T01:30:15.645 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi193/crash 2024-09-16T01:30:15.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash -- . 2024-09-16T01:30:15.681 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-16T01:30:15.681 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-09-16T01:30:15.682 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-16T01:30:15.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-16T01:30:15.780 INFO:tasks.cephadm:Compressing logs... 2024-09-16T01:30:15.780 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:30:15.782 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:30:15.785 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:30:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T01:30:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-16T01:30:15.817 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-16T01:30:15.817 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-16T01:30:15.817 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-16T01:30:15.818 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-16T01:30:15.819 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T01:30:15.820 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 86.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T01:30:15.820 INFO:teuthology.orchestra.run.smithi078.stderr: 94.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-16T01:30:15.820 INFO:teuthology.orchestra.run.smithi078.stderr: 94.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-16T01:30:15.821 INFO:teuthology.orchestra.run.smithi078.stderr: 94.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-16T01:30:15.821 INFO:teuthology.orchestra.run.smithi078.stderr: 94.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-16T01:30:15.822 INFO:teuthology.orchestra.run.smithi148.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T01:30:15.822 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-16T01:30:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-16T01:30:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-16T01:30:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T01:30:15.823 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-16T01:30:15.825 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-16T01:30:15.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T01:30:15.827 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-16T01:30:15.828 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-16T01:30:15.828 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-16T01:30:15.828 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.i.log/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log: 2024-09-16T01:30:15.828 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-16T01:30:15.828 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 94.4% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log 2024-09-16T01:30:15.831 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-16T01:30:15.832 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-16T01:30:15.832 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log: /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-16T01:30:15.832 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-16T01:30:15.832 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-16T01:30:15.832 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-16T01:30:15.833 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-16T01:30:15.907 INFO:teuthology.orchestra.run.smithi148.stderr: 82.3% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T01:30:15.917 INFO:teuthology.orchestra.run.smithi193.stderr: 82.3% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T01:30:15.923 INFO:teuthology.orchestra.run.smithi193.stderr: 92.3% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-16T01:30:15.942 INFO:teuthology.orchestra.run.smithi148.stderr: 90.4% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T01:30:15.968 INFO:teuthology.orchestra.run.smithi148.stderr: 86.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T01:30:15.987 INFO:teuthology.orchestra.run.smithi078.stderr: 90.3% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T01:30:16.001 INFO:teuthology.orchestra.run.smithi148.stderr: 92.2% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-16T01:30:16.029 INFO:teuthology.orchestra.run.smithi148.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T01:30:16.039 INFO:teuthology.orchestra.run.smithi193.stderr: 90.9% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T01:30:16.040 INFO:teuthology.orchestra.run.smithi078.stderr: 84.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-16T01:30:16.112 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T01:30:16.125 INFO:teuthology.orchestra.run.smithi193.stderr: 90.4% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-16T01:30:16.125 INFO:teuthology.orchestra.run.smithi193.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T01:30:16.131 INFO:teuthology.orchestra.run.smithi193.stderr: 86.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-16T01:30:16.170 INFO:teuthology.orchestra.run.smithi078.stderr: 94.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-16T01:30:16.404 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-16T01:30:16.983 INFO:teuthology.orchestra.run.smithi193.stderr: 94.4% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-16T01:30:17.127 INFO:teuthology.orchestra.run.smithi193.stderr: 94.0% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-16T01:30:18.094 INFO:teuthology.orchestra.run.smithi078.stderr: 90.5% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-16T01:30:19.475 INFO:teuthology.orchestra.run.smithi148.stderr: 94.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-16T01:30:19.821 INFO:teuthology.orchestra.run.smithi148.stderr: 91.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-16T01:30:22.497 INFO:teuthology.orchestra.run.smithi193.stderr: 93.5% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-16T01:30:25.492 INFO:teuthology.orchestra.run.smithi193.stderr: 91.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-16T01:30:27.281 INFO:teuthology.orchestra.run.smithi148.stderr: 93.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-16T01:30:29.838 INFO:teuthology.orchestra.run.smithi078.stderr: 90.2% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-16T01:31:32.164 INFO:teuthology.orchestra.run.smithi148.stderr: 95.2% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-16T01:33:08.833 INFO:teuthology.orchestra.run.smithi193.stderr: 90.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-16T01:34:34.104 INFO:teuthology.orchestra.run.smithi193.stderr: 90.8% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-16T01:35:37.517 INFO:teuthology.orchestra.run.smithi078.stderr: 90.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-16T01:36:04.265 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-16T01:36:07.211 INFO:teuthology.orchestra.run.smithi193.stderr: 90.9% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-16T01:37:26.750 INFO:teuthology.orchestra.run.smithi193.stderr: 90.5% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-16T01:37:26.751 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-09-16T01:37:26.752 INFO:teuthology.orchestra.run.smithi193.stderr:real 7m10.952s 2024-09-16T01:37:26.752 INFO:teuthology.orchestra.run.smithi193.stderr:user 5m23.345s 2024-09-16T01:37:26.752 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m12.198s 2024-09-16T01:37:40.527 INFO:teuthology.orchestra.run.smithi148.stderr: 90.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-16T01:38:19.137 INFO:teuthology.orchestra.run.smithi148.stderr: 90.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-16T01:38:56.876 INFO:teuthology.orchestra.run.smithi148.stderr: 91.0% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-16T01:38:56.878 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T01:38:56.878 INFO:teuthology.orchestra.run.smithi148.stderr:real 8m41.081s 2024-09-16T01:38:56.878 INFO:teuthology.orchestra.run.smithi148.stderr:user 5m58.452s 2024-09-16T01:38:56.878 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m14.164s 2024-09-16T01:39:14.670 INFO:teuthology.orchestra.run.smithi078.stderr: 90.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-16T01:40:40.069 INFO:teuthology.orchestra.run.smithi078.stderr: 90.7% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-16T01:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr: 90.6% -- replaced with /var/log/ceph/fa4731de-73c6-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-16T01:41:23.618 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-16T01:41:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:real 11m7.822s 2024-09-16T01:41:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:user 6m53.986s 2024-09-16T01:41:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m16.704s 2024-09-16T01:41:23.619 INFO:tasks.cephadm:Archiving logs... 2024-09-16T01:41:23.620 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi078/log 2024-09-16T01:41:23.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:42:37.479 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi148/log 2024-09-16T01:42:37.480 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:43:42.291 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi193/log 2024-09-16T01:43:42.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:44:40.814 INFO:tasks.cephadm:Removing cluster... 2024-09-16T01:44:40.839 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force 2024-09-16T01:44:41.042 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:44:41.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force 2024-09-16T01:44:42.041 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:44:42.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa4731de-73c6-11ef-bceb-c7b262605968 --force 2024-09-16T01:44:42.945 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: fa4731de-73c6-11ef-bceb-c7b262605968 2024-09-16T01:44:43.585 INFO:tasks.cephadm:Removing cephadm ... 2024-09-16T01:44:43.585 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:44:43.605 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:44:43.634 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-16T01:44:43.650 INFO:tasks.cephadm:Teardown complete 2024-09-16T01:44:43.651 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T01:44:43.662 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T01:44:43.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:44:43.665 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:44:43.677 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:44:43.794 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T01:44:43.808 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T01:44:43.808 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:44:43.811 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:44:43.814 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:44:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:44:43.828 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:44:43.829 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:44:43.842 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:44:43.843 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-09-16T01:44:43.843 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 -439us[ -440us] +/- 42ms 2024-09-16T01:44:43.843 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +260us[ +259us] +/- 66ms 2024-09-16T01:44:43.843 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +516us[ +515us] +/- 78ms 2024-09-16T01:44:43.843 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:44:43.849 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:44:43.849 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-16T01:44:43.849 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -439us[ -433us] +/- 42ms 2024-09-16T01:44:43.850 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +268us[ +268us] +/- 66ms 2024-09-16T01:44:43.850 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +504us[ +510us] +/- 78ms 2024-09-16T01:44:43.850 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:44:43.908 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:44:43.908 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-09-16T01:44:43.909 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 -446us[ -447us] +/- 42ms 2024-09-16T01:44:43.909 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 +247us[ +245us] +/- 66ms 2024-09-16T01:44:43.909 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +500us[ +499us] +/- 78ms 2024-09-16T01:44:43.909 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:44:43.909 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T01:44:43.919 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T01:44:43.920 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T01:44:43.985 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:44:44.148 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:44:44.300 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:44:44.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:44:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:44.445 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:44:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:44.479 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:44:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:44.508 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T01:44:44.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T01:44:44.530 INFO:teuthology.task.internal:Duration was 3047.437191 seconds 2024-09-16T01:44:44.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T01:44:44.541 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T01:44:44.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:44:44.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:44:44.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:44:44.612 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:44:44.612 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:44:44.949 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T01:44:44.950 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-09-16T01:44:44.950 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:44:44.973 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T01:44:44.973 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:44:45.005 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-09-16T01:44:45.006 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:44:45.039 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T01:44:45.040 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:44:45.045 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:44:45.049 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:44:45.109 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T01:44:45.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:44:45.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:44:45.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:44:45.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T01:44:46.434 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T01:44:46.435 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:44:46.467 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:44:46.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:44:46.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T01:44:47.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.700 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-09-16T01:44:47.701 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-09-16T01:44:47.824 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-09-16T01:44:47.840 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:47.857 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:47.876 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:44:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:47.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T01:44:48.290 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T01:44:48.291 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi078 2024-09-16T01:44:48.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:44:48.463 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi148 2024-09-16T01:44:48.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:44:48.506 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906435/remote/smithi193 2024-09-16T01:44:48.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:44:48.551 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T01:44:48.551 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:44:48.554 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:44:48.558 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:44:48.602 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T01:44:48.616 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T01:44:48.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T01:44:48.676 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T01:44:48.676 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:44:48.679 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:44:48.682 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:44:48.694 INFO:teuthology.orchestra.run.smithi078.stdout: 265641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:44 /home/ubuntu/cephtest 2024-09-16T01:44:48.697 INFO:teuthology.orchestra.run.smithi148.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:44 /home/ubuntu/cephtest 2024-09-16T01:44:48.700 INFO:teuthology.orchestra.run.smithi193.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:44 /home/ubuntu/cephtest 2024-09-16T01:44:48.717 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T01:44:48.730 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T01:44:48.762 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 3047.4371905326843 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T01:44:48.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:44:48.846 INFO:teuthology.run:pass